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ORGANIZATION 


RTO  is  the  single  focus  in  NATO  for  Defence  Research  and  Technology  activities.  Its  mission  is  to  conduct  and  promote 
co-operative  research  and  information  exchange.  The  objective  is  to  support  the  development  and  effective  use  of 
national  defence  research  and  technology  and  to  meet  the  military  needs  of  the  Alliance,  to  maintain  a  technological 
lead,  and  to  provide  advice  to  NATO  and  national  decision  makers.  The  RTO  performs  its  mission  with  the  support  of  an 
extensive  network  of  national  experts.  It  also  ensures  effective  co-ordination  with  other  NATO  bodies  involved  in  R&T 
activities. 


RTO  reports  both  to  the  Military  Committee  of  NATO  and  to  the  Conference  of  National  Armament  Directors. 
It  comprises  a  Research  and  Technology  Board  (RTB)  as  the  highest  level  of  national  representation  and  the  Research 
and  Technology  Agency  (RTA),  a  dedicated  staff  with  its  headquarters  in  Neuilly,  near  Paris,  France.  In  order  to 
facilitate  contacts  with  the  military  users  and  other  NATO  activities,  a  small  part  of  the  RTA  staff  is  located  in  NATO 
Headquarters  in  Brussels.  The  Brussels  staff  also  co-ordinates  RTO’s  co-operation  with  nations  in  Middle  and  Eastern 
Europe,  to  which  RTO  attaches  particular  importance  especially  as  working  together  in  the  field  of  research  is  one  of  the 
more  promising  areas  of  co-operation. 


The  total  spectrum  of  R&T  activities  is  covered  by  the  following  7 
•  AVT  Applied  Vehicle  Technology  Panel 
Human  Factors  and  Medicine  Panel 
Information  Systems  Technology  Panel 
NMSG  NATO  Modelling  and  Simulation  Group 
SAS  System  Analysis  and  Studies  Panel 

Systems  Concepts  and  Integration  Panel 
Sensors  and  Electronics  Technology  Panel 


HFM 

1ST 


SCI 

SET 


bodies: 


These  bodies  are  made  up  of  national  representatives  as  well  as  generally  recognised  ‘world  class’  scientists.  They  also 
provide  a  communication  link  to  military  users  and  other  NATO  bodies.  RTO’s  scientific  and  technological  work  is 
carried  out  by  Technical  Teams,  created  for  specific  activities  and  with  a  specific  duration.  Such  Technical  Teams  can 
organise  workshops,  symposia,  field  trials,  lecture  series  and  training  courses.  An  important  function  of  these  Technical 
Teams  is  to  ensure  the  continuity  of  the  expert  networks. 


RTO  builds  upon  earlier  co-operation  in  defence  research  and  technology  as  set-up  under  the  Advisory  Group  for 
Aerospace  Research  and  Development  (AGARD)  and  the  Defence  Research  Group  (DRG).  AGARD  and  the  DRG  share 
common  roots  in  that  they  were  both  established  at  the  initiative  of  Dr  Theodore  von  Karman,  a  leading  aerospace 
scientist,  who  early  on  recognised  the  importance  of  scientific  support  for  the  Allied  Armed  Forces.  RTO  is  capitalising 
on  these  common  roots  in  order  to  provide  the  Alliance  and  the  NATO  nations  with  a  strong  scientific  and  technological 
basis  that  will  guarantee  a  solid  base  for  the  future. 


The  content  of  this  publication  has  been  reproduced 
directly  from  material  supplied  by  RTO  or  the  authors. 


Published  April  2012 

Copyright  ©  RTO/NATO  2012 
All  Rights  Reserved 
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Single  copies  of  this  publication  or  of  a  part  of  it  may  be  made  for  individual  use  only.  The  approval  of  the  RTA 
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Acceptability  Criteria:  A  set  of  standards  that  a  particular  model  or  simulation  and  its  associated  data  must  meet 
to  be  acceptable  for  its  intended  use. 

Acceptance:  The  decision  to  use  the  results  produced  by  a  model  or  simulation  and  its  associated  data  for  an 
intended  use. 

Activity:  A  set  of  tasks  that  consumes  time  and  resources  and  whose  performance  is  necessary  for  the  execution 
of  the  VV&A  processes. 

Accreditation:  The  official  certification  that  a  model  or  simulation  and  its  associated  data  are  acceptable  for  use 
for  a  specific  purpose. 

Development  Product:  Any  artifact  of  the  M&S  development  or  modification  processes. 

M&S  Use  Risk:  The  risk  incurred  by  the  application  of  M&S  results  and  that  is  calculated  as  a  function  of 
M&S  uncertainty,  W&A  assessment  uncertainty,  and  the  importance  of  the  M&S  results  based  on  the  intended 
use. 

Risk:  A  measure  of  the  probability  and  severity  of  undesired  effects  often  taken  as  the  simple  product  of  probability 
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Validation:  The  process  of  determining  the  degree  to  which  a  model  or  simulation  and  its  associated  data  are 
an  accurate  representation  of  the  real  world  from  the  perspective  of  the  intended  uses  of  the  model. 

Verification:  The  process  of  determining  that  a  model  or  simulation  and  its  associated  data  accurately  represent 
the  developer’s  conceptual  description  and  specifications. 
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Risk-Based  Tailoring  of  the  Verification,  Validation, 
and  Accreditation/Acceptance  Processes 

(RTO-TR-MSG-054) 

Executive  Summary 

In  September  2006,  the  RTO  approved  the  formation  of  MSG-054/TG-037,  “An  Overlay  Standard  for 
Verification,  Validation,  and  Accreditation  (VV&A)  of  Federations”.  This  M&S  Group  was  established  to 
formalize  the  draft  recommended  practice  previously  produced  by  MSG-019  as  an  international  industry 
standard  by  participating  in  vetting  the  document  through  the  Simulation  Interoperability  Standards 
Organization’s  (SISO)  and  the  Institute  of  Electrical  and  Electronics  Engineers  Standards  Association’s 
(IEEE-SA)  standards  processes.  Additionally,  the  group  was  tasked  to  address  the  concept  of  VV&A 
tailoring  and  to  provide  sufficient  guidance  on  the  tailoring  and  practical  implementation  of  the  VV&A 
overlay. 

MSG-054  worked  in  conjunction  with  the  SISO  VV&A  Product  Development  Group  (PDG)  to  adjudicate 
the  comments  from  two  rounds  of  review  of  the  MSG-019  product  and  to  revise  the  product  to  reflect  the 
comments.  On  the  recommendation  of  the  SISO  VV&A  PDG,  the  IEEE-SA  Standards  Board  approved  the 
project  “P 151 6.4  -  Verification,  Validation,  and  Accreditation  of  a  Federation,  an  Overlay  to  the  High 
Level  Architecture  Federation  Development  and  Execution  Process”  on  December  6,  2006.  A  balloting 
group  was  formed  and  formal  balloting  was  conducted.  As  in  the  previous  phase  of  the  process,  MSG-054 
supported  the  adjudication  of  reviewer  comments.  As  a  result  of  MSG-054/TG-037  efforts,  IEEE  Std 
1516.4™-2007  was  approved  by  the  IEEE-SA  Standards  Board  in  September  2007  and  was  published  as 
an  international  industry  standard  in  December  2007. 

In  addition  to  the  establishment  of  the  IEEE  standard,  MSG-054/TG-037  worked  to  develop  the  components 
of  tailoring  guidance.  The  resulting  product,  the  Verification  and  Validation  (V&V)  Composite  Model 
[see  Appendix  1  which  describes  the  components  of  the  V&V  processes  (i.e.,  phases,  activities,  and  tasks)] 
from  which  to  select  in  order  to  match  the  risk  and  resource  constraints  of  the  V&V  efforts  while  still  adhering 
to  relevant  policies,  standards,  and  guidance.  The  V&V  Composite  Model  is  a  superset  of  the  possible 
activities  and  the  context  in  which  those  activities  can  be  tailored  into  working  V&V  processes. 

This  report  describes  the  accomplishments  of  MSG-054/TG-037. 


RTO-TR-MSG-054 


ES  - 1 


Adaptation  fondee  sur  le  risque,  des  processus 
de  verification,  de  validation,  et 
d’accreditation/d’ acceptation 

(RTO-TR-MSG-054) 


Synthese 

En  septembre  2006,  la  RTO  a  approuve  la  creation  du  MSG-054/TG-037,  «  Une  norme  Overlay  pour  la 
Verification,  la  Validation,  et  l’Accreditation  (W&A)  des  Federations  ».  Ce  groupe  M&S  a  ete  mis  en  place 
pour  formaliser  le  projet  de  technique  recommandee  foumi  anterieurement  par  le  MSG-019  comme  norme 
industrielle  intemationale,  en  revisant  ce  document  au  travers  du  processus  de  normalisation  de  1’  Organisation 
des  Normes  de  Simulation  pour  l’lnteroperabilite  (SISO)  et  de  l’Association  des  Normes  de  l’institut  des 
Ingenieurs  Electriciens  et  Electroniciens  (IEEE-SA).  Le  groupe  a  egalement  eu  pour  tache  de  traiter  le 
concept  d’adaptation  du  VV&A  et  de  foumir  des  directives  suffisantes  pour  l’adaptation  et  la  mise  en  oeuvre 
pratique  de  F  overlay  VV&A. 

Le  MSG-054  a  travaille  en  relation  avec  le  Groupe  de  Developpement  de  Produits  (PDG)  SISO  VV&A 
pour  se  prononcer  sur  les  observations  issues  des  deux  phases  d’examen  du  produit  MSG-019  et  pour 
remanier  le  produit  afin  de  prendre  en  compte  ces  observations.  Sur  recommandation  du  PDG  SISO 
VV&A,  la  commission  des  normes  de  1’IEEE-SA  a  approuve  le  projet  «  P1516.4  -  Verification,  Validation, 
et  Accreditation  d’une  Federation,  et  d’un  Overlay  pour  le  processus  de  developpement  et  d’execution  de 
la  federation  d’ architecture  de  haut  niveau  »  le  6  decembre  2006.  Un  groupe  de  scrutateurs  a  ete  forme  et 
un  scrutin  formel  a  ete  organise.  Comme  pour  la  phase  anterieure  du  processus,  le  MSG-054  a  soutenu 
l’agrement  des  resultats  des  rapporteurs.  A  la  suite  des  travaux  du  MSG-054/TG-037,  la  Norme  IEEE 
1516.4  ™-2007  a  ete  approuvee  par  la  commission  des  normes  de  1’IEEE-SA  en  septembre  2007  et  a  ete 
publiee  en  tant  que  norme  industrielle  intemationale  en  decembre  2007. 

En  complement  a  la  mise  en  place  d’une  norme  IEEE,  le  MSG-054/TG-037  a  travaille  a  developper  les 
composantes  des  directives  d’adaptation.  Le  produit  resultant  est  le  Modele  Composite  de  Verification  et 
de  Validation  (V&V)  [voir  l’Appendice  1  qui  decrit  les  composants  des  processus  V&V  (par  exemple,  les 
phases,  les  activites,  et  les  taches)]  a  partir  duquel  sont  operes  des  choix  pour  faire  face  aux  risques  et  aux 
contraintes  de  ressources  des  activites  V&V  tout  en  continuant  a  se  conformer  aux  politiques,  normes, 
et  directives  pertinentes.  Le  Modele  Composite  V&V  est  un  sur-ensemble  d’activites  possibles  et  aussi  le 
contexte  dans  lequel  ces  activites  peuvent  etre  adaptees  dans  les  processus  de  travail  de  V&V. 

Ce  rapport  decrit  les  realisations  du  MSG-054/TG-037. 
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RISK-BASED  TAILORING  OF  THE  VERIFICATION,  VALIDATION, 
AND  ACCREDITATION/ACCEPTANCE  PROCESSES 

1.0  INTRODUCTION 

1.1  Background 

The  idea  for  an  exploratory  team  on  Verification,  Validation,  and  Accreditation  (VV&A)  of  federations  was  first 
proposed  during  the  5th  North  Atlantic  Treaty  Organisation  (NATO)  Modelling  and  Simulation  Group  meeting 
in  July  2000.  Initial  discussions  over  the  Internet  and  during  a  meeting  in  September  2000  by  interested  parties  at 
the  Defense  Modeling  and  Simulation  Office  in  the  United  States  made  clear  that  sufficient  interest  and 
substance  existed  for  the  formation  of  a  NATO  Task  Group  (TG)  on  the  subject  of  federation  VV&A. 

MSG-0 19/TG-0 16  was  established  in  2001  and  met  several  times  through  2005  in  both  the  United  States  and 
Europe.  MSG-019  carefully  analyzed  IEEE  Std  1516.3™-2003  [l]1  to  identify  where  VV&A  contributions 
were  needed,  but  were  undefined.  The  Task  Group  also  developed  an  initial  model  of  the  VV&A  process  to 
ensure  the  development  of  consistent  VV&A  guidance  for  federations. 

In  January  2005,  the  Simulation  Interoperability  Standards  Organization  (SISO)  established  the  Verification, 
Validation,  and  Accreditation  Overlay  to  the  Federation  Development  and  Execution  Process  (FEDEP)  Product 
Development  Group  (PDG)  to  produce  a  W&A  Overlay  to  IEEE  1516.3™.  At  the  same  time,  MSG-019  also 
had  determined  the  need  for  such  a  standard  and  began  contributing  to  the  development  as  part  of  the  SISO 
VV&A  Overlay  PDG.  MSG-019  ensured  the  standard  under  development  would  meet  NATO/Partners  for  Peace 
(PfP)  specific  needs. 

MSG-019  completed  its  objectives  by  producing  a  draft  recommended  practice  for  the  VV&A  of  federations 
[2].  MSG-019  reported  its  activities  in  the  Research  and  Technology  Organisation  (RTO)  Technical  Report, 
Verification,  Validation,  and  Accreditation  (VV&A)  of  Federations  (TR-MSG-019)  [3]  and  closed  in  September 
2006.  MSG-019  made  these  recommendations: 

•  NATO  should  adopt  the  draft  recommended  practice  as  an  interim  Standardization  Agreement 
(STANAG)  until  the  IEEE  standard  is  issued  and  then  revise  the  STANAG  to  adopt  the  IEEE  standard. 

•  MSG-054  (the  follow-on  study  group)  should: 

•  Participate  in  the  IEEE  Std  1516.4™-2007  [4]  balloting  process; 

•  Monitor  on-going  VV&A  case  study  initiative  and  incorporate  suggested  improvements  to  the 
Overlay  during  the  balloting  process;  and 

•  Study  the  relationship  between  user  risk,  acceptance  criteria,  and  tailoring  of  the  VV&A  process. 

•  NATO  should  consider  adopting  similar  collaborative  relationships  with  international  standards  bodies 
as  appropriate. 

In  September  2006,  the  RTO  approved  the  formation  of  MSG-054/TG-037,  “An  Overlay  Standard  for 
Verification,  Validation,  and  Accreditation  (VV&A)  of  Federations”,  to  formalize  the  draft  recommended 
practice  previously  produced  by  MSG-0 1 9  as  an  international  industry  standard  by  participating  in  vetting  the 


1  A  list  of  References  may  be  found  in  Section  5. 
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document  through  SISO’s  and  the  Institute  of  Electrical  and  Electronics  Engineers  Standards  Association’s 
(IEEE-SA)  standards  processes.  The  specific  tasking  assigned  to  MSG-054/TG-037  is  discussed  next  in 
Section  1.2. 

1.2  Tasking 

The  Terms  of  Reference  [5]  for  MSG-054/TG-037  identified  these  specific  activities  to  be  performed: 

•  Complete  development  of  a  canonical  VV&A  process  started  by  MSG-019; 

•  Identify  a  set  of  terms  needed  to  adequately  describe  the  elements  of  that  process  and  choose  consistent 
definitions  for  those  terms; 

•  Apply  the  canonical  VV&A  process  to  define  the  activities,  tasks,  activity  inputs  and  activity  products 
of  the  FEDEP  VV&A  Overlay; 

•  Participate  in  the  review,  comment  and  balloting  efforts  of  the  SISO  VV&A  Overlay  PDG  that  will 
make  the  VV&A  Overlay  an  IEEE  standard; 

•  Address  such  special  topics  as  basic  concepts  and  VV&A  tailoring  and  provide  sufficient  guidance  in 
these  areas  to  practically  implement  the  VV&A  overlay; 

•  Develop  qualifications  that  guide  application  of  the  VV&A  Overlay  standard  to  NATO/PfP-specific 
initiatives; 

•  Develop  the  standard  adoption  and  qualification  guidance  recommendations  for  a  final  Technical 
Report;  and 

•  Finalize  the  Technical  Report. 

1.3  Task  Group  Membership 

Nations  participating  in  the  MSG-054/TG-037  were  Canada,  France,  Germany,  the  United  Kingdom,  and  the 
United  States.  The  United  States  representative  served  as  the  chairperson. 

1.4  Scope  of  Effort 

The  MSG-019  product  defined  a  comprehensive  set  of  VV&A  activities,  tasks,  and  products  that  overlay  the 
phases  of  the  FEDEP.  As  expected  in  an  overlay,  the  product  adopted  the  fundamental  principles  associated 
with  the  parent  document  including  the  concept  of  tailoring.  The  FEDEP  specifically  stated  that  the  development 
and  execution  process  was  meant  to  be  tailored  for  a  specific  application.  The  concept  of  tailoring  was  extended 
to  VV&A  by  including  a  high  level  discussion  regarding  tailoring  and  its  impact  on  planning  for  and 
implementing  VV&A. 

During  the  balloting  rounds,  there  were  a  significant  number  of  comments  that  related  to  the  defined  tailoring 
section.  Although  there  was  general  recognition  of  the  critical  nature  of  the  tailoring  function,  several  of  the 
balloting  comments  identified  the  need  for  additional,  detailed  guidance  on  how  and  when  to  tailor  the 
verification  and  validation  processes.  The  SISO  VV&A  PDG  Drafting  Group  intended  that  additional,  more 
detailed  guidance  products  would  be  developed  to  address  key  concepts  such  as  tailoring.  Tailoring  VV&A  to 
address  a  specific  intended  use  is  driven  by  the  risk  associated  with  the  use  of  the  federation  results  and 
resource  constraints  such  as  available  information,  budget,  skills,  and  time.  Quantifying  risk  and  the  tolerance 
for  those  risks  are  factors  that  drive  tailoring  decisions.  Developing  a  tailoring  approach  is  predicated  on  an 
understanding  of  the  relationship  and  impacts  of  tailoring  and  risk. 
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Section  2  describes  the  results  of  the  IEEE-SA  standards  process  that  resulted  in  an  approved  international 
industry  standard.  Section  3  describes  the  V&V  Composite  Model  which  was  developed  as  guidance  for 
tailoring  VV&A  processes. 


2.0  IEEE  STD  1516.4™-2007 

The  SISO  VV&A  PDG  conducted  the  first  round  of  review  and  comment  of  the  MSG-019  product  from 
March  31  to  May  15,  2006,  and  a  total  of  214  comments  were  received.  The  SISO  W&A  PDG  Drafting  Group 
and  MSG-054/TG-037  met  June  5-9,  2006  in  Ottawa,  Ontario,  Canada,  to  review  and  finalize  resolutions  for 
non-editorial  type  comments.  A  SISO  VV&A  PDG  teleconference  was  conducted  on  June  28,  2006  to  vote  on 
the  resolutions  of  the  comments.  The  vote  was  conducted  in  accordance  with  the  PDG’s  voting  rules. 
One  comment  that  was  unresolved  during  the  teleconference  was  resolved  via  an  electronic  vote  for  which  the 
polling  feature  of  the  SISO  VV&A  PDG  discussion  forum  was  used. 

A  second  round  of  comments  by  the  SISO  VV&A  PDG  was  conducted  from  August  4  to  September  4,  2006. 
An  additional  non-editorial  type  22  comments  were  addressed  by  the  SISO  VV&A  PDG  at  its  meeting  on 
September  14,  2006  held  in  conjunction  with  the  2006  Fall  Simulation  Interoperability  Workshop.  As  a  result 
of  voting  conducted  in  accordance  with  the  PDG’s  voting  rules,  two  comments  were  withdrawn  by  the 
submitter  and  20  resolutions  were  approved.  The  PDG  also  voted  to  recommend  the  SISO  Standards  Activity 
Committee  approve  the  proposed  standard  for  entry  into  the  IEEE-SA  standards  process. 

A  Project  Authorization  Request  was  submitted  to  the  IEEE-SA  Standards  Board  to  begin  that  organization’s 
standards  process.  On  December  6,  2006  the  IEEE-SA  Standards  Board  approved  the  project  “PI 5 1 6.4  - 
Verification,  Validation,  and  Accreditation  of  a  Federation,  an  Overlay  to  the  High  Level  Architecture 
Federation  Development  and  Execution  Process.”  Both  SISO  and  IEEE  distributed  messages  announcing  the 
project  and  inviting  members  to  join  the  ballot  pool  in  January  2007. 

There  were  55  individuals  who  joined  the  Sponsor  Ballot  that  was  conducted  from  April  17  through  May  17, 
2006.  The  Sponsor  Ballot  achieved  a  93.6%  (44/47  responders)  approval  rate.  Table  1  summarizes  how  the 
votes  were  distributed  according  to  the  IEEE-SA  classification  codes. 


Table  1:  Sponsor  Ballot  Voter  Summary. 


Classification 

Affirmative 

Negative 

Abstain 

Unreturned 

Total 

Academic 

5 

0 

0 

0 

5 

General  Interest 

11 

0 

3 

2 

16 

Government/Military 

12 

0 

1 

0 

13 

Producer 

5 

1 

1 

0 

7 

User 

11 

2 

0 

1 

14 

Totals 

44 

3 

5 

3 

55 

A  total  of  80  comments  were  received.  All  comments  were  addressed  and  negative  comments  were  adjudicated 
directly  with  the  submitter.  A  revised  version  of  draft  P1516.4  was  produced  and  a  message  announcing  the 
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recirculation  ballot  was  distributed  on  June  26,  2007.  The  recirculation  ballot  ended  on  July  6,  2007  with  no  new 
comments  received. 

As  a  result  of  the  recirculation  ballot,  one  negative  vote  was  changed  to  affirmative.  The  document  achieved  a 
96%  (45/47  respondents)  affirmative  vote.  Table  2  summarizes  the  results  of  the  recirculation  ballot. 


Table  2:  Recirculation  Ballot  Voter  Summary. 


Classification 

Affirmative 

Negative 

Abstain 

Unreturned 

Total 

Academic 

5 

0 

0 

0 

5 

General  Interest 

11 

0 

3 

2 

16 

Government/Military 

12 

0 

1 

0 

13 

Producer 

5 

1 

1 

0 

7 

User 

12 

1 

0 

1 

14 

Totals 

45 

2 

5 

3 

55 

As  a  result  of  MSG-054/TG-037  efforts,  IEEE  Std  1516.4™-2007  [4]  was  approved  by  the  IEEE-SA 
Standards  Board  in  September  2007  and  was  published  as  an  international  industry  standard  in  December  2007. 


3.0  DESCRIPTION  OF  V&V  COMPOSITE  MODEL 

3.1  Approach  to  W&A  Tailoring 

Enabling  the  tailoring  of  VV&A  processes  begins  with  describing  the  range  of  phases,  activities,  and  tasks 
that  possibly  could  be  performed  when  implementing  VV&A  processes.  The  efforts  of  MSG-054/TG-037 
resulted  in  the  description  of  the  phases,  activities,  and  tasks  for  the  Verification  and  Validation  (V&V)  processes 
only.  The  phases,  activities,  and  tasks  for  the  accreditation  process  were  not  completed  as  a  result  of  these 
efforts. 

The  V&V  Composite  Model  (see  Appendix  1)  describes  the  components  of  the  processes  (i.e.,  phases,  activities, 
and  tasks)  from  which  to  select  to  match  the  risk  and  resource  constraints  of  the  V&V  efforts  while  still 
adhering  to  relevant  policies,  standards,  and  guidance.  The  V&V  Composite  Model  is  a  superset  of  the  possible 
activities  and  the  context  in  which  those  activities  can  be  tailored  into  working  V&V  processes. 

Several  assumptions  underlie  the  construction  of  the  V&V  Composite  Model: 

•  The  V&V  Composite  Model  is  a  general  model  representing  V&V  processes. 

•  The  audience  for  the  V&V  Composite  Model  are  stakeholders  in  the  V&V  processes. 

•  Accreditation  refers  to  the  process  used  to  arrive  at  an  acceptability  decision  based  on  an  assessment 
of  evidence  collected  by  implementing  V&V  processes. 

•  Acceptability  refers  to  the  decision  to  apply  an  M&S  for  an  intended  use. 

•  Acceptability  criteria  adequately  describe  the  M&S  capabilities  needed  to  support  an  intended  use. 
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*  The  primary  purpose  for  implementing  V&V  processes  is  to  collect  evidence  to  support  applying 
M&S  results  for  an  intended  use. 

•  Risk  refers  to  the  risk  of  using  an  M&S  (i.e.,  use  risk). 

The  paper  included  as  Appendix  2,  “Decomposing  the  W&A  Processes  to  Support  Their  Tailoring”  (08S-SIW- 
066)  was  presented  at  the  2008  Spring  Simulation  Interoperability  Workshop  sponsored  by  the  Simulation 
Interoperability  Standards  Organization  in  Providence,  Rhode  Island,  USA,  on  April  14-18,  2008  [6].  Authored 
by  the  United  States,  the  paper  describes  the  components  of  the  V&V  Composite  Model  phases,  activities,  and 
tasks.  Since  2008,  the  V&V  Composite  Model  has  been  updated  and  revised,  but  the  paper  still  serves  as  a 
good  source  of  background  information.  Additionally,  information  from  the  paper  was  adapted  for  inclusion 
in  Sections  3.2  and  3.3. 

3.2  Components  of  Tailoring  Guidance 

Using  risk  as  an  effective  tailoring  mechanism  for  VV&A  processes  depends  on  understanding  the  factors  that 
drive  the  risk  as  well  as  the  VV&A  activities  and  tasks  that  can  and  should  be  implemented  to  address  the  use 
risks.  The  V&V  Composite  Model  (see  Appendix  1)  describes  one  perspective  of  the  activities  and  tasks  that 
constitute  the  V&V  processes  for  M&S  upon  which  to  found  risk-based  VV&A  tailoring. 

The  V&V  Composite  Model  generalizes  the  guidance  provided  in  IEEE  Std  1516.4™-2007  [4]  to  apply  to 
M&S  beyond  just  federations,  and  extends  that  guidance  to  consider  the  broad  range  of  possible  sources  of 
validation  evidence.  This  model  of  the  V&V  processes  defines  the  tasks  that  may  be  tailored  to  best  suit  the 
limits  of  acceptable  use  risk  and  optimize  the  resources  available  for  VV&A. 

As  the  complexity  of  the  M&S  and  the  associated  V&V  processes  increase  while  budgets  and  schedules 
decrease,  it  is  likely  that  not  all  V&V  activities  and  tasks  will  be  able  to  be  performed.  Additionally,  when  V&V 
is  performed  on  legacy  M&S,  legacy  developmental  products  may  not  be  available  to  review,  analyze,  and  test. 
To  avoid  becoming  overwhelmed  by  the  tasks  described,  to  utilize  resources  efficiently,  and  to  mitigate  risk 
effectively,  tailoring  of  the  V&V  processes  is  needed.  Tailoring  guidance  helps  determine  the  activities  required 
for  different  V&V  implementations. 

Tailoring  guidance  is  embedded  within  the  descriptions  of  the  activities  in  the  V&V  Composite  Model. 
This  guidance  helps  the  V&V  practitioner  determine  what  tasks  to  omit  and  to  understand  the  corresponding 
risk. 


3.3  Overview  of  the  V&V  Composite  Model 

The  V&V  Composite  Model  comprises  phases,  activities,  and  tasks  that  decompose  the  general  V&V  processes. 
The  highest  levels  are  phases  and  there  are  eight  phases  in  the  V&V  Composite  Model: 


•  PHASE  1:  Plan  the  V&V  Effort.  The  V&V  processes  are  planned  to  address  the  acceptability 
criteria  (developed  during  planning  for  accreditation)  and  the  requirements  for  using  the  M&S. 
The  V&V  practitioner  executes  and  evolves  the  V&V  Plan  throughout  the  V&V  effort.  This  includes 
monitoring  the  V&V  effort  and  adjusting  the  V&V  Plan  to  better  reflect  any  new  information  gained 
while  executing  the  V&V  processes.  The  V&V  Plan  should  be  approved  by  the  individual  responsible 
for  ensuring  the  V&V  processes  are  implemented. 

•  PHASE  2:  Apply  Relevant  Historical  Information.  When  a  legacy  M&S  is  to  be  applied  for  the  same 
or  a  similar  intended  use  for  which  it  has  been  previously  accredited,  considerable  historical  information 
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may  exist  upon  which  to  base  the  new  accreditation  decision.  In  those  instances,  it  is  possible  the  entire 
V&V  effort  could  focus  upon  the  collection  and  analysis  of  historical  information.  When  a  legacy  M&S 
is  changed,  when  the  simuland  changes,  or  when  the  intended  use  changes,  historical  information  may 
contribute  to  identifying  the  gaps  in  the  additional  V&V  evidence  that  will  be  needed.  Applying 
historical  information  begins  by  determining  how  closely  the  prior  intended  uses  match  the  current 
intended  use  to  determine  what  part  of  the  history  is  relevant  to  the  current  problem.  Historical 
information  can  come  from  prior  V&V  and  testing  activities,  the  records  of  prior  uses,  and  the 
developer’s  accounts  of  the  M&S  capabilities  and  limitations.  The  V&V  practitioner  should  analyze  the 
relevant  historical  record  to  identify  the  factors  that  may  constrain  use.  Then  a  coherent  picture  of  the 
M&S  capabilities  and  limitations  based  upon  the  historical  evidence  can  be  assembled.  If  results 
validation  will  be  performed,  the  V&V  practitioner  can  use  the  historical  evidence  to  identify  possible 
output  sampling  areas  to  improve  its  efficiency. 

•  PHASE  3:  Verify  and  Validate  the  Conceptual  Model.  The  conceptual  model  serves  as  a  bridge 
between  the  requirements  for  developing  an  M&S  and  the  M&S  design,  providing  the  developer’s 
interpretation  of  the  requirements.  The  conceptual  model  is  the  documented  theoretical  approach  to 
the  design  of  the  M&S.  The  conceptual  model  describes  what  the  M&S  is  expected  to  do,  to  be, 
and  what  data  and  other  elements  are  needed  for  a  successful  outcome.  The  V&V  practitioner  begins 
by  characterizing  its  coverage  of  the  intended  use  and  inferring  the  M&S  capabilities  from  the 
information  that  the  conceptual  model  contains.  Conceptual  model  verification  involves  checking  it 
for  internal  consistency  problems.  After  that,  the  V&V  practitioner  can  evaluate  the  conceptual  model 
against  the  acceptability  criteria  to  determine  its  validity.  The  objective  of  performing  conceptual 
model  validation  is  to  demonstrate  that  the  M&S  functional  elements  accurately  and  completely 
represent  the  M&S  requirements  and  to  identify  where  assumptions,  limitations,  or  architectural 
structure  impact  the  intended  M&S  use.  If  available,  the  V&V  practitioner  should  also  verify  and, 
if  needed,  validate  the  use  scenarios. 

•  PHASE  4:  Perform  Supplemental  Verification.  How  much  the  V&V  practitioner  contributes  to 
development  product  verification  depends  largely  upon  the  verification  activities  performed  and 
documented  by  the  developer  of  the  M&S.  The  V&V  practitioner  begins  the  supplemental  verification 
by  collecting  the  developer’s  development  products  and  determining  how  much  additional  verification 
is  needed  to  achieve  the  desired  confidence.  The  types  of  verification  tasks  that  can  be  performed 
depend  largely  upon  the  development  products  available.  The  verification  and  validation  activities 
performed  and  documented  by  the  developer  against  the  requirements  for  building  the  M&S  can  be  used 
to  support  the  M&S  validation  activities  performed  by  the  V&V  practitioner.  The  V&V  practitioner 
should  leverage  as  much  of  the  developer’s  verification  and  validation  evidence  as  possible  and  only 
perform  what  supplemental  verification  is  needed  to  increase  the  confidence  in  and  bolster  the 
validation  evidence. 

•  PHASE  5:  Apply  The  Verification  Products  to  Validation.  During  this  phase,  the  V&V  practitioner 
applies  the  collected  verification  information  to  develop  evidence  on  the  M&S  validity,  adds  to  the 
information  to  support  the  accreditation  recommendations,  estimates  the  coverage  and  uncertainties  in 
the  development  product  verification  evidence,  and  employs  the  verification  evidence  to  identify 
factors  that  may  constrain  use  and  candidate  output  sampling  areas. 

•  PHASE  6:  Verify  and  Validate  the  Data  and  Knowledge  Sets.  V&V  is  performed  on  both  the  M&S 
and  data  used  by  the  M&S.  This  phase  focuses  upon  the  data  used  to  build  the  M&S,  the  data  used 
as  input  into  the  M&S,  and  the  data  output  by  the  M&S.  Data  and  knowledge  sets  are  distinguished 
by  definition.  Knowledge  is  a  class  of  data  that  incorporates  a  knowledge  representation  scheme 
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(e.g.,  production  rules,  semantic  network),  while  data  are  representations  of  facts,  concepts, 
or  instructions  in  a  formalized  manner  suitable  for  communication,  interpretation,  or  processing  by 
humans  or  by  automatic  means.  Data  and  knowledge  set  V&V  should  be  performed  because  of  then 
independence  from  the  M&S  and  because  the  M&S  relies  upon  those  sets  to  produce  its  output. 
The  phase  begins  by  identifying  their  sources  and  the  pedigrees  of  those  sources  because  in  many  cases 
the  organizations  responsible  for  producing  the  data  differ  from  the  organizations  that  either  develop  or 
use  the  M&S.  The  credibility  of  those  data-producing  organizations  often  weighs  heavily  in  determining 
the  credibility  of  the  data  itself.  After  establishing  the  pedigrees  of  the  data  and  knowledge,  the  V&V 
practitioner  should  verify  the  internal  consistency  of  the  data  sets;  verify  any  transformations  used 
to  make  the  data  accessible  and  meaningful  to  the  M&S  (e.g.,  units  transformations,  coordinate 
transformations)  then  validate  the  completeness  and  correctness  of  the  data  sets  against  the  acceptability 
criteria.  The  data  validation  information  can  then  be  used  to  identify  output  sampling  guidance  and 
factors  that  may  constrain  M&S  use. 

•  PHASE  7:  Validate  the  M&S  Results.  During  this  phase,  the  V&V  practitioner  reviews,  analyzes, 
and  tests  the  M&S.  Tests  are  run  to  produce  output;  the  output  is  analyzed  to  determine  capabilities; 
and  the  capabilities  are  reviewed  against  the  acceptability  criteria  to  infer  validity.  The  V&V 
practitioner  uses  the  developer’s  test  results  to  tailor  the  results  validation  effort,  if  those  results  are 
available.  Design  of  experiments  techniques  are  applied  to  build  the  test  cases  used  to  sample  the 
M&S  output.  The  V&V  practitioner  executes  the  test  cases,  collects  the  output,  analyzes  the  output, 
and  documents  an  integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected 
V&V  evidence. 

•  PHASE  8:  Integrate  the  V&V  Evidence.  The  preceding  phases  produce  the  evidence  that  this  phase 
integrates  into  a  description  of  the  M&S  validity  to  support  the  accreditation  recommendations. 
The  V&V  practitioner  examines  the  evidence  produced  by  the  preceding  activities,  forms  a  consistent 
picture  of  the  M&S  validity  and  estimates  the  confidence  in  that  determination.  The  V&V  practitioner 
identifies  the  factors  that  may  constrain  use  and  combines  those  constraints  with  the  assessments  of 
the  M&S  completeness  and  correctness  for  the  intended  use.  Then,  the  V&V  practitioner  assembles 
the  results  of  this  integration  with  the  evidence  produced  by  the  prior  V&V  activities  into  the  V&V 
Report.  The  information  in  the  V&V  Report  should  present  the  evidence  upon  which  an  accreditation 
assessment  can  be  made  to  base  the  accreditation  recommendations.  This  phase  may  include  any  post¬ 
execution  follow-up  and  archival  activities. 

Figure  1  identifies  the  activities  that  compose  each  of  the  eight  phases.  Each  activity  includes  a  description, 
as  well  as  possible  tailoring  guidance,  notes,  and  assumptions.  Each  activity  is  decomposed  into  tasks. 
Each  task  includes  a  description,  information  required  to  perform  the  task,  the  source  of  that  information, 
and  the  information  produced  by  the  task.  The  V&V  Composite  Model  assumes  that  once  a  product  is  created, 
it  is  available  for  all  subsequent  activities;  even  though  the  product  might  not  be  identified  as  information  that 
is  required  to  perform  the  task. 
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Figure  1:  Activities  Associated  with  Each  V&V  Composite  Model  Phase. 


Although  many  of  the  activities  and  tasks  in  Figure  1  might  appear  sequential,  the  intention  is  not  to  restrict 
the  implementation  to  any  specific  approach  (e.g.,  waterfall,  spiral,  evolutionary).  Rather,  the  diagram  is 
meant  to  highlight  the  general  activities  from  which  V&V  processes  can  be  tailored  to  meet  the  needs  of  the 
M&S  application  based  upon  the  intended  use. 


3.4  Verification  of  the  V&V  Composite  Model 

The  Web  Ontology  Language  (OWL)  [7]  was  used  to  create  a  partial  ontological  implementation  of  the 
Risk-based  VV&A  methodology  structure  to  verify  its  composition  and  relationships.  In  the  future, 
a  complete  ontological  implementation  could  serve  as  a  tool  to  promote  coherent  tailoring,  to  generate  reliable 
documentation,  and  to  facilitate  future  methodology  upgrades.  A  few  detailed  examples  were  implemented 
using  OWL  to  demonstrate  the  basic  implementation  principles;  however  the  RBA  methodology  is  not  yet 
finalized,  and  neither  are  the  details  of  the  relationships  of  the  individual  activities  and  tasks.  The  MSG-054 
German  representative  developed  a  paper  that  describes  the  results  of  this  effort  (Appendix  3). 
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Three  conclusions  were  reached  based  on  the  work  thus  far  to  verify  the  V&V  Composite  Model: 

1)  The  implementation  of  the  RBA  methodology  as  an  OWL  ontology  is  feasible  without  making  any 
compromise  and  appears  to  promise  a  greater  flexibility  and  upgradeability  than  more  conventional 
approaches  while  preserving  the  fundamental  advantages  of  the  RBA  methodology. 

2)  It  will  allow,  with  some  additional  work,  a  seamless  integration  in  comprehensive  ontology-based 
modeling  and  simulation  tool  platforms. 

3)  Ontological  tools  implementing  reasoning  mechanisms  potentially  can  track  tailoring  effects  throughout 
the  RBA  sequence  of  activities  and  tasks;  additional  work  might  reveal  whether  sufficient  quality 
assurance  guarantees  can  be  provided  in  this  manner  for  deeper,  more  customized  tailoring  approaches. 
Further  work  might  also  consider  using  such  tools  to  visualize  tailoring  effects  in  a  user  friendly 
manner  in  order  to  facilitate  tailoring  decisions  and  to  explore  the  relationships  between  risk 
constellations  and  the  tailoring  strategies  most  appropriate  for  them. 

4.0  CONCLUSIONS  AND  RECOMMENDATIONS 

MSG-054/TG-037  succeeded  in  its  task  to  institutionalize  VV&A  processes  overlaid  upon  the  federation 
development  and  execution  process  with  the  publication  of  IEEE  Std  1516.4™-2007  [4]  in  December  2007. 
As  an  international  commercial  standard,  IEEE  Std  1516.4™-2007  is  available  for  application  by  government, 
industry,  and  academia  worldwide. 

IEEE  Std  151 6.4™-2007  focused  on  VV&A  processes  for  federations  and  purposely  did  not  address  the  W&A 
processes  associated  with  the  individual  federates  (e.g.,  federation  managers,  data  collectors,  live  systems,  M&S, 
viewers,  etc.).  Because  information  from  the  VV&A  processes  conducted  on  federates  is  used  by  the  federation, 
it  is  important  that  the  federate  level  information  be  complete,  credible,  and  accessible.  Therefore,  MSG-054/ 
TG-037  leveraged  the  IEEE  Std  1516.4™-2007  concepts  to  develop  a  composite  model  of  the  V&V  processes 
that  could  be  applied  to  federation  and  federate  level  M&S,  as  well  as  to  M&S  that  were  not  distributed. 
MSG-054/TG-037  accomplishments  further  extended  to  the  development  of  guidance  for  tailoring  the  W&A 
processes. 

MSG-054/TG-037  was  successful  in  producing  a  draft  composite  model  of  the  V&V  processes  but  was  unable 
to  complete  work  on  the  accreditation/acceptance  processes  due  to  the  availability  of  resources  to  the  various 
members.  Additionally,  while  the  members  of  MSG-054/TG-037  were  able  conduct  some  level  of  vetting  of 
the  V&V  Composite  Model  within  their  national  M&S  communities,  the  model  would  benefit  from  more 
extensive  vetting. 

MSG-054/TG-037  leveraged  NATO  STANAG  4603  [8]  and  IEEE  Std  1516.3™-2003  [1]  in  the  development 
of  IEEE  Std  1516.4™-2007  [4]  and  the  V&V  Composite  Model.  The  group  found  that  leveraging  established 
standards  ensured  consistency  and  understanding  of  basic  VV&A  concepts  across  products. 

MSG-054/TG-037  feels  it  made  progress  in  providing  detailed  descriptions  of  how  to  conduct  V&V  processes, 
but  the  work  is  not  complete. 

Based  on  these  conclusions,  MSG-054/TG-037  makes  the  following  recommendations: 

•  NATO  should  process  and  ratify  IEEE  Std  1516.4™-2007,  IEEE  Recommended  Practice  for 
Verification,  Validation,  and  Accreditation  of  a  Federation  -  An  Overlay  to  the  High  Level  Architecture 
Federation  Development  and  Execution  Process,  as  a  NATO  Standardization  Agreement. 
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•  NATO  should  continue  the  decompose  the  accreditation/acceptance  aspects  of  the  V&V  Composite 
Model  necessary  to  evolve  it  into  a  complete  VV&A  Composite  model.  Enhancements  include: 

•  Update  the  V&V  activities  and  tasks  by  including  more  information  in  Activity  Tailoring  Guidance, 
Activity  Notes,  and  Activity  Assumptions. 

•  Incorporate  accreditation/acceptance  activities  and  tasks. 

•  Revise  based  on  comments  received  from  reviews. 

•  Add  phase  descriptions. 

•  Map  the  completed  VV&A  Composite  Model  to  IEEE  Std  15 16.4™-2007. 

•  Member  Nations  should  each  to  continue  vetting  the  composite  model  within  their  M&S  communities. 

•  NATO  should  complete  the  effort  to  create  an  ontological  implementation  of  the  completed  VV&A 
Composite  Model  to  verify  its  consistency,  completeness,  and  correctness. 

•  NATO  should  continue  participation  in  the  Simulation  Interoperability  Standards  Organization  and  lead 
the  effort  to  process  the  VV&A  Composite  Model  as  a  standard  once  it  is  completed  and  fully  vetted. 
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TABLE  OF  CONTENTS 
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Execute  and  Evolve  the  V&V  Plan 

PHASE  2 

APPLY  RELEVANT  HISTORICAL  INFORMATION 
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Collect  and  Analyze  Developer  Accounts  of  M&S  Capabilities  and  Limitations 

Activity  2.2 

Collect  and  Analyze  the  V&V  and  Testing  Histories 

Activity  2.3 

Collect  and  Analyze  the  Prior  Use  History 

Activity  2.4 
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PHASE  3 

VERIFY  AND  VALIDATE  THE  M&S  CONCEPTUAL  MODEL 

Activity  3.1 

Verify  the  Conceptual  Model 
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Validate  the  Conceptual  Model 

Activity  3.3 

Verify  and  Validate  the  Use  Scenarios 

Activity  3.4 

Integrate  and  Employ  the  Conceptual  Model  V&V  Evidence 

PHASE  4 

PERFORM  SUPPLEMENTAL  VERIFICATION 

Activity  4. 1 

Determine  the  Scope  of  Supplemental  Verification  Needed 
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Verify  the  M&S  Design  Products 
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Verify  the  M&S  Object  Code 

Activity  4.4 

Verify  the  M&S  Executable 
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Verify  the  Development  Products  for  Standards  Compliance 

Activity  4.6 

Integrate  the  Supplemental  Verification  Evidence 

PHASE  5 

APPLY  THE  VERIFICATION  PRODUCTS  TO  VALIDATION 

Activity  5.1 

Determine  the  Effects  of  Development  Product  Inconsistencies 

Activity  5.2 

Evaluate  M&S  Validity  from  the  Verification  Products 

Activity  5.3 

Employ  the  Verification  Evidence 

PHASE  6 

VERIFY  AND  VALIDATE  THE  DATA  AND  KNOWLEDGE  SETS 

Activity  6.1 

Identify  the  Data  and  Knowledge  Sources  and  Their  Pedigrees 

Activity  6.2 

Verify  the  Data  and  Knowledge  Sets 

Activity  6.3 

Validate  the  Data  and  Knowledge  Sets  Where  Needed 

Activity  6.4 

Integrate  and  Employ  the  Data  and  Knowledge  V&V  Evidence 
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# 

Title 

PHASE  7 

VALIDATE  THE  M&S  RESULTS 

Activity  7. 1 

Leverage  the  Developer  Test  Results  for  Results  Validation 

Activity  7.2 

Plan  for  Results  Validation 

Activity  7.3 

Collect  M&S  Output  for  Results  Validation 

Activity  7.4 

Integrate  and  Employ  the  Validation  Testing  Results 

PHASE  8 

INTEGRATE  THE  V&V  EVIDENCE 

Activity  8.1 

Integrate  the  V&V  Evidence 

Activity  8.2 

Derive  the  Conclusions  from  the  Integrated  V&V  Evidence 

Activity  8.3 

Prepare  the  V&V  Report 

Activity  8.4 

Support  Any  Archival  of  the  V&V  Products 

Al.l  COMPOSITE  MODEL  OVERVIEW 

The  Composite  Model  defines  the  superset  of  V&V  activities  from  which  a  V&V  Plan  would  be  tailored. 
Each  activity  in  the  Composite  Model  is  described  in  a  consistent  format.  This  format  includes: 

•  The  activity  description; 

•  Tailoring  guidance  associated  with  the  activity; 

•  Notes  related  that  add  clarification  to  the  purpose  or  implementation  of  the  activity; 

•  Assumptions  that  underlie  the  implementation  of  the  activity;  and 

•  Descriptions  of  the  tasks  to  be  performed: 

•  Descriptions  of  the  information  required  to  perform  each  task;  and 

•  Descriptions  of  the  information  produced  by  each  task. 

Instances  where  tailoring  guidance,  notes  or  assumptions  are  not  defined  for  a  specific  activity  are  denoted  by 
(Blank). 
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A1.2  PHASE  1  -  PLAN  THE  V&V  EFFORT 
Activity  1.1  -  Develop  the  V&V  Approach 

Activity  Description:  This  activity  selects  the  specific  activities  and  tasks  that  will  be  performed  as  part  of 
the  V&V  approach.  It  also  identifies  the  techniques  and  tools  needed  to  accomplish  the  selected  V&V  tasks. 
Together,  these  elements  define  the  V&V  approach  form  which  the  rest  of  the  V&V  Plan  is  constructed. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  The  term  “complexity”  is  used  here,  in  the  most  general  sense,  as  a  measure  of  the  difficulty 
of  a  task.  These  tasks  do  not  include  any  iteration  either  within  this  activity  or  between  the  other  V&V  Planning 
activities. 

Activity  Assumptions:  Guidance  exists  to  aid  the  selection  of  V&V  tasks  and  techniques  and  needs  prioritization. 

Task  Descriptions: 

1.1.1  Obtain  the  Acceptability  Criteria  Report  and  Accreditation  Plan,  if  available. 

Information  Required: 

•  Accreditation  Plan;  Acceptability  Criteria  Report  (Source:  Preceding  accreditation  activity) 
Information  Produced: 

•  Accreditation  Plan;  Acceptability  Criteria  Report  are  forwarded  to  follow-on  activities 

1.1.2  Obtain  the  M&S  requirements. 

Information  Required: 

•  M&S  requirements  (Source:  User) 

Information  Produced: 

•  M&S  requirements  are  forwarded  to  follow-on  activities 

1.1.3  Obtain  a  description  of  the  M&S. 

Information  Required: 

•  M&S  description  (Source:  Developer) 

Information  Produced: 

•  M&S  description  are  forwarded  to  follow-on  activities 

1.1.4  Analyze  the  M&S  requirements,  accreditation  information  requirements  and  M&S  description  to 
develop  the  scope  statement  for  the  V&V  effort. 

Information  Required: 

•  M&S  requirements  (Source:  Task  1.1.2) 

•  M&S  description  (Source:  Task  1.1.3) 

•  Accreditation  information  requirements  (Source:  preceding  accreditation  activity) 
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Information  Produced: 

•  V&V  effort  scope  statement 

1.1.5  Analyze  the  Acceptability  Criteria  Report  or  the  M&S  Requirements  and  M&S  description  to  determine 
the  acceptability  criteria  that  define  the  required  functionality. 

Information  Required: 

•  Acceptability  Criteria  Report  or  M&S  requirements  (Source:  Task  1.1.1  or  1.1.2) 

•  M&S  description  (Source:  Task  1.1.3) 

Information  Produced: 

•  Acceptability  criteria  defining  needed  functionality 

1.1.6  Analyze  the  Acceptability  Criteria  Report  or  the  M&S  Requirements  and  M&S  description  to  determine 
the  acceptability  criteria  that  define  the  required  quality. 

Information  Required: 

•  Acceptability  Criteria  Report  or  M&S  Requirements  (Source:  Task  1.1.1  or  1.1.2) 

•  M&S  description  (Source:  Task  1.1.3) 

Information  Produced: 

•  Acceptability  criteria  defining  needed  quality 

1.1.7  Analyze  the  Accreditation  Plan,  the  acceptability  criteria  and  any  VV&A  task  selection  guidance  to 
determine  the  V&V  activities  and  tasks  that  could  be  performed. 

Information  Required: 

•  Accreditation  Plan  (Source:  Preceding  accreditation  activity) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Task  1.1.5) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Task  1.1.6) 

•  VV&A  task  selection  guidance  (Source:  Open  literature) 

Information  Produced: 

•  Candidate  V&V  activities  and  tasks 

1.1.8  Analyze  the  acceptability  criteria,  the  Accreditation  Plan  and  the  needs  prioritization  guidance  to  assign 
the  needs  priorities  to  the  acceptability  criteria. 

Information  Required: 

•  Accreditation  Plan  (Source:  Preceding  accreditation  activity) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Task  1.1.5) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Task  1.1.6) 

•  Needs  prioritization  guidance  (Source:  Open  literature) 

Information  Produced: 

•  Assigned  acceptability  criteria  priorities 
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1.1.9  Analyze  the  candidate  V&V  activities  and  tasks,  the  level  of  technique  rigor  required,  the  acceptability 
criteria  and  any  VV&A  technique  selection  guidance  to  select  the  V&V  techniques  that  could  be 
applied. 

Information  Required 

•  Candidate  V&V  activities  and  tasks  (Source:  Task  1.1.7) 

•  Assigned  acceptability  criteria  priorities  (Source:  Task  1.1.8) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Task  1.1.5) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Task  1.1.6) 

•  VV&A  technique  selection  guidance  (Source:  Open  literature) 

Information  Produced: 

•  Candidate  V&V  techniques 

1.1.10  Analyze  the  candidate  V&V  techniques  to  identify  the  tools  needed  to  support  the  application  of  those 
techniques. 

Information  Required: 

•  Candidate  V&V  techniques  (Source:  Task  1.1.9) 

Information  Produced: 

•  Candidate  tools  needed  to  support  V&V 

1.1.11  Analyze  the  Accreditation  Plan,  the  acceptability  criteria  and  aspects  of  the  V&V  approach  to  estimate 
the  complexity  of  each  of  the  V&V  tasks  to  be  performed. 

Information  Required: 

•  Accreditation  Plan  (Source:  Preceding  accreditation  activity) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Task  1.1.5) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Task  1.1.6) 

•  Candidate  V&V  activities  and  tasks  (Source:  Task  1.1.7) 

•  Candidate  V&V  techniques  (Source:  Task  1.1.9) 

•  Candidate  tools  needed  to  support  V&V  (Source:  Task  1.1.10) 

Information  Produced: 

•  Estimated  complexity  of  each  of  the  V&V  tasks  to  be  performed 

1.1.12  Assemble  the  candidate  V&V  tasks,  techniques,  tools  and  estimated  complexity  into  a  description  of 
the  V&V  approach. 

Information  Required: 

•  Candidate  V&V  activities  and  tasks  (Source:  Task  1.1.7) 

•  Candidate  V&V  techniques  (Source:  Task  1.1.9) 

•  Candidate  tools  needed  to  support  V&V  (Source:  Task  1.1.10) 

•  Estimated  complexity  of  each  of  the  V&V  tasks  to  be  performed  (Source:  1.1.11) 

Information  Produced 

•  Description  of  the  V&V  approach 
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Activity  1.2  -  Identify  the  Required  V&V  Resources 

Activity  Description:  This  activity  identifies  the  candidate  personnel  resources  needed  to  perforin  the  V&V 
activities.  These  candidates  are  selected  from  the  skills  required  to  perform  the  V&V  tasks  and  the  personnel 
that  are  available  to  the  V&V  effort. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes: 

•  These  tasks  do  not  include  any  iteration  either  within  this  activity  or  between  the  other  V&V  Planning 
activities;  and 

•  The  term  “labor  estimate”  refers  to  the  number  of  hours  required. 

Activity  Assumptions:  The  Sponsor  can  provide  a  list  of  the  available  candidate  V&V  personnel.  If  the  Sponsor 
does  not  provide  that  information  then  the  V&V  practitioner  may  need  to  identify  those  personnel. 

Task  Descriptions: 

1 .2. 1  Analyze  the  V&V  approach  to  determine  the  skills  required  to  perform  each  of  the  tasks. 

Information  Required: 

•  V&V  approach  (Source:  Activity  1.1) 

Information  Produced: 

•  Skills  required  for  each  V&V  task  to  be  performed 

1.2.2  Obtain  a  list  of  the  available  candidate  V&V  personnel  using  the  skills  required  for  each  V&V  task  to 
be  performed. 

Information  Required: 

•  List  of  the  available  candidate  V&V  personnel  (Source:  Sponsor) 

•  Skills  required  for  each  V&V  task  to  be  performed  (Source:  Task  1.2.1) 

Information  Produced: 

•  List  of  the  available  candidate  V&V  personnel  are  forwarded  to  follow-on  activities 

1.2.3  Analyze  the  V&V  approach  and  the  skills  required  for  each  V&V  task  to  be  performed  to  estimate  the 
amount  of  labor  required  to  perform  each  task  for  each  skill. 

Information  Required: 

•  V&V  approach  (Source:  Activity  1.1) 

•  Skills  required  for  each  V&V  task  to  be  performed  (Source:  Task  1.2.1) 

Information  Produced: 

•  Labor  estimate  for  each  task  and  skill 

1.2.4  Analyze  the  V&V  approach,  the  list  of  available  candidate  V&V  personnel  and  the  labor  estimate  for 
each  task  and  skill  to  identify  candidate  personnel  to  perform  each  of  the  V&V  tasks. 
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Information  Required: 

•  V&V  approach  (Source:  Activity  1.1) 

•  List  of  the  available  candidate  V&V  personnel  (Source:  Task  1.2.2) 

•  Labor  estimate  for  each  task  and  skill  (Source:  Task  1.2.3) 

Information  Produced: 

•  Candidate  personnel  to  perform  each  of  the  V&V  tasks 

1.2.5  Analyze  the  labor  estimate  for  each  task  and  skill  and  the  candidate  personnel  to  perform  each  of  the 
V&V  tasks  to  map  the  labor  required  onto  each  candidate. 

Information  Required: 

•  Labor  estimate  for  each  task  and  skill  (Source:  Task  1.2.3) 

•  Candidate  personnel  to  perform  each  of  the  V&V  tasks  (Source:  Task  1.2.4) 

Information  Produced: 

•  Labor  required  from  each  candidate  person 

1.2.6  Assemble  the  skills  required,  available  candidate  personnel,  labor  estimates,  candidate  personnel  task 
assignments  and  labor  from  each  candidate  into  the  required  V&V  resources. 

Information  Required: 

•  Skills  required  for  each  V&V  task  to  be  performed  (Source:  Task  1.2.1) 

•  List  of  the  available  candidate  V&V  personnel  (Source:  Task  1.2.2) 

•  Labor  estimate  for  each  task  and  skill  (Source:  Task  1.2.3) 

•  Candidate  personnel  to  perform  each  of  the  V&V  tasks  (Source:  Task  1.2.4) 

•  Labor  required  from  each  candidate  person  (Source:  Task  1.2.5) 

Information  Produced: 

•  Required  V&V  resources 

Activity  1.3  -  Build  the  V&V  Schedule 

Activity  Description:  This  activity  produces  a  schedule  for  the  V&V  effort  that  aligns  with  the  accreditation 
schedule  and,  ultimately,  the  project  schedule. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes: 

•  This  set  of  tasks  implicitly  assumes  that  the  schedule  can  determine  the  costs  rather  than  vice  versa. 

•  These  tasks  do  not  include  any  iteration  either  within  this  activity  or  between  the  other  V&V  Planning 
activities. 

•  The  Sponsor  can  provide  a  schedule  of  the  availability  of  the  candidate  V&V  personnel.  If  the  Sponsor 
cannot  provide  that  information  then  the  V&V  practitioner  may  need  to  obtain  it. 

Activity  Assumptions:  The  accreditation  plan  describes  the  programmatic  scheduling  constraints  (in  the 
accreditation  activities). 
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Task  Descriptions: 

1.3.1  Obtain  the  availability  schedule  of  the  candidate  V&V  personnel  using  the  required  V&V  resources. 
Information  Required: 

•  Candidate  V&V  personnel  availability  schedule  (Source:  Sponsor) 

•  Required  V&V  resources  (Source:  Activity  1.2) 

Information  Produced: 

•  Candidate  V&V  personnel  availability  schedule  is  forwarded  to  follow-on  activities 

1.3.2  Analyze  the  V&V  approach  to  determine  the  task  dependencies. 

Information  Required: 

•  V&V  approach  (Source:  Activity  1.1) 

Information  Produced: 

•  V&V  task  dependencies 

1.3.3  Analyze  the  V&V  approach  and  the  V&V  task  dependencies  to  determine  the  V&V  task  sequencing. 

Information  Required: 

•  V&V  approach  (Source:  Activity  1.1) 

•  V&V  task  dependencies  (Source:  Task  1.3.2) 

Information  Produced: 

•  V&V  task  sequencing 

1.3.4  Analyze  the  Accreditation  Plan,  the  candidate  V&V  personnel  availability  schedule  and  the  V&V  task 
sequencing  to  determine  the  V&V  schedule. 

Information  Required: 

•  Accreditation  Plan  (Source:  Preceding  accreditation  activity) 

•  Candidate  V&V  personnel  availability  schedule  (Source:  Task  1.3.1) 

•  V&V  task  sequencing  (Source:  Task  1.3.3) 

Information  Produced: 

•  V&V  effort  schedule 

Activity  1.4  -  Estimate  the  V&V  Costs 

Activity  Description:  This  activity  produces  estimates  of  the  costs  of  performing  V&V.  These  costs  include 
components  for  labor,  travel  and  costs  other  than  labor  and  travel  (e.g.,  to  purchase  tools  or  hardware). 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  These  tasks  do  not  include  any  iteration  either  within  this  activity  or  between  the  other  V&V 
Planning  activities. 

Activity  Assumptions:  (Blank) 
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Task  Descriptions: 

1.4.1  Analyze  the  V&V  approach  and  the  required  V&V  resources  to  estimate  the  labor  costs  for  each 
V&V  task. 

Information  Required: 

•  V&V  approach  (Source:  Activity  1.1) 

•  Required  V&V  resources  (Source:  Activity  1.2) 

Information  Produced: 

•  Estimated  labor  costs  for  each  V&V  task 

1.4.2  Analyze  the  V&V  approach  and  the  required  V&V  resources  to  estimate  the  possible  travel  required 
to  perform  V&V  and  their  costs. 

Information  Required: 

•  V&V  approach  (Source:  Activity  1.1) 

•  Required  V&V  resources  (Source:  Activity  1.2) 

Information  Produced: 

•  Estimated  V&V  travel  requirements  and  costs 

1.4.3  Analyze  the  estimated  labor  costs  for  each  V&V  task  and  the  estimated  V&V  travel  requirements  and 
costs  to  estimate  the  costs  other  than  labor  and  travel  required  to  perform  V&V. 

Information  Required: 

•  Estimated  labor  costs  for  each  V&V  task  (Source:  Task  1.4.1) 

•  Estimated  V&V  travel  requirements  and  costs  (Source:  Task  1.4.2) 

Information  Produced: 

•  Estimated  V&V  costs  other  than  labor  and  travel 

1.4.4  Analyze  the  V&V  approach,  the  estimated  labor  costs  for  each  V&V  task,  the  estimated  V&V  travel 
requirements  and  costs  and  the  estimated  V&V  costs  other  than  labor  and  travel  to  estimate  a  total 
cost  of  the  V&V  effort. 

Information  Required: 

•  V&V  approach  (Source:  Activity  1.1) 

•  Estimated  labor  costs  for  each  V&V  task  (Source:  Task  1.4.1) 

•  Estimated  V&V  travel  requirements  and  costs  (Source:  Task  31.4.2) 

•  Estimated  V&V  costs  other  than  labor  and  travel  (Source:  Task  1.4.3) 

Information  Produced: 

•  Estimated  V&V  effort  cost 

Activity  1.5  -  Prepare  the  V&V  Plan 

Activity  Description:  This  activity  assembles  the  components  produced  by  the  previous  activities  into  the 
V&V  Plan,  circulates  that  plan  to  interested  parties,  and  responds  to  the  comments  from  those  parties  with 
modifications  to  the  plan. 
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ORGANIZATION 


Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes: 

•  Iteration  between  the  various  parts  of  plan  preparation  can  be  coordinated  within  this  activity;  and 

•  Modifying  the  plan  may  require  revisiting  any  of  the  preceding  V&V  Planning  activities. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

1.5.1  Construct  the  initial  requirements  traceability  matrix. 

Information  Required 

•  M&S  requirements  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

Information  Produced: 

•  Initial  requirements  traceability  matrix 

1.5.2  Prepare  citations  of  the  information  used  for  V&V  Planning. 

Information  Required: 

•  Accreditation  Plan  (Source:  Activity  1.1) 

•  Acceptability  Criteria  Report  (Source:  Activity  1.1) 

•  M&S  requirements  (Source:  Activity  1.1) 

•  M&S  description  (Source:  Activity  1.1) 

•  List  of  the  available  candidate  V&V  personnel  (Source:  Activity  1.2) 

•  Candidate  V&V  personnel  availability  schedule  (Source:  Activity  1.3) 

Information  Produced: 

•  Citations  of  information  used  for  V&V  Planning 

1.5.3  Assemble  the  V&V  approach,  V&V  resource  needs,  V&V  schedule,  V&V  cost  estimate  and  citations 
of  the  information  used  for  V&V  Planning  into  an  integrated  draft  V&V  Plan. 

Information  Required: 

•  V&V  approach  (Source:  Activity  1.1) 

•  Required  V&V  resources  (Source:  Activity  1.2) 

•  V&V  schedule  (Source:  Activity  1.3) 

•  Estimated  V&V  effort  cost  (Source:  Activity  1.4) 

•  Initial  requirements  traceability  matrix  (Source:  Task  1.5.1) 

•  Citations  of  information  used  for  V&V  Planning  (Source:  Task  1.5.2) 

Information  Produced: 

•  Draft  V&V  Plan 

1.5.4  Analyze  the  draft  V&V  Plan  for  internal  consistency  and  correct  any  inconsistencies. 
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Information  Required: 

•  Draft  V&V  Plan  (Source:  Task  1.5.3) 

Information  Produced: 

•  Correct  draft  V&V  Plan 

1.5.5  Circulate  the  corrected  draft  V&V  Plan  for  review  and  collect  comments  and  recommendations  for 
change. 

Information  Required: 

•  Corrected  draft  V&V  Plan  (Source:  Task  1.5.4) 

Information  Produced: 

•  Comments  and  recommendations  for  change  received  on  draft  V&V  Plan 

1.5.6  Modify  the  corrected  draft  V&V  Plan  from  the  comments  and  recommendations  for  change  received. 
Information  Required: 

•  Corrected  draft  V&V  Plan  (Source:  Task  1.5.4) 

•  Comments  and  recommendations  for  change  received  on  draft  V&V  Plan  (Source:  Task  1.5.5) 

Information  Produced: 

•  Modified  V&V  Plan 

1.5.7  Analyze  the  modified  V&V  Plan  for  internal  consistency,  correct  any  inconsistencies  and  prepare  the 
final  V&V  Plan. 

Information  Required: 

•  Modified  V&V  Plan  (Source:  Task  1.5.6) 

Information  Produced: 

•  Final  V&V  Plan 

1.5.8  Prepare  the  V&V  Plan  versioning  history  with  the  final  V&V  Plan. 

Information  Required: 

•  Final  V&V  Plan  (Source:  Task  1.5.7) 

Information  Produced: 

•  V&V  Plan  versioning  history 


Activity  1.6  -  Execute  and  Evolve  the  V&V  Plan 

Activity  Description:  This  activity  executes  the  V&V  Plan,  monitors  the  results  of  the  execution  process  and 
evolves  the  V&V  Plan  in  response  to  those  results. 

Activity  Tailoring  Guidance:  (Blank) 
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ORGANIZATION 


Activity  Notes:  Developing  the  V&V  Plan  may  require  several  iterations  and  necessitate  revisiting  the 
preceding  planning  activities.  Determining  what  parts  of  the  plan  need  to  be  evolved  will  be  done  within  this 
activity. 

Activity  Assumptions:  A  V&V  Plan  versioning  history  is  kept,  possibly  as  part  of  a  program-level  configuration 
management  process. 

Task  Descriptions: 

1.6.1  Execute  the  V&V  Plan. 

Information  Required: 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Updated  V&V  Plan 

1.6.2  Monitor  the  V&V  Plan  execution  and  collect  data  on  the  results  of  that  execution. 

Information  Required: 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  V&V  Plan  execution  results 

1.6.3  Analyze  the  V&V  Plan  and  the  results  from  its  execution  to  determine  the  deviation  from  the  expected 
results. 

Information  Required: 

•  V&V  Plan  execution  results  (Source:  Task  1.6.2) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Deviations  from  the  expected  V&V  Plan  results 

1.6.4  Analyze  the  revised  Accreditation  Plan  to  determine  the  modifications  needed  to  accommodate  those 
revisions  in  the  V&V  Plan. 

Information  Required: 

•  Revised  Accreditation  Plan  (Source:  Preceding  accreditation  activity) 

Information  Produced: 

•  Modifications  to  the  existing  V&V  Plan  from  Accreditation  Plan  revisions 

1.6.5  Obtain  changes  to  the  M&S  requirements. 

Information  Required: 

•  M&S  requirements  changes  (Source:  User) 
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Information  Produced: 

•  M&S  requirements  changes  forwarded  to  follow-on  activities 

1.6.6  Analyze  changes  to  the  M&S  requirements  to  determine  modifications  needed  to  the  requirements 
traceability  matrix. 

Information  Required: 

•  M&S  requirements  (Source:  Task  1.6.5) 

Information  Produced: 

•  Requirements  traceability  matrix  modifications 

1.6.7  Analyze  the  revisions  to  the  Accreditation  Plan  and  deviations  from  the  expected  V&V  Plan  results  to 
determine  the  modifications  needed  to  accommodate  the  observed  deviations  from  the  existing  V&V 
Plan. 

Information  Required: 

•  Deviations  from  the  expected  V&V  Plan  results  (Source:  Task  1.6.3) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Modifications  to  the  existing  V&V  Plan  from  deviations  from  that  plan 

1.6.8  Analyze  the  revisions  to  the  Accreditation  Plan  and  deviations  from  the  expected  V&V  Plan  results  to 
determine  the  modifications  needed  to  accommodate  the  observed  deviations  from  the  existing  V&V 
Plan. 

Information  Required: 

•  Modifications  to  the  existing  V&V  Plan  from  Accreditation  Plan  revisions  (Source:  Task  1.6.4) 

•  Modifications  to  the  existing  V&V  Plan  from  deviations  from  that  plan  (Source:  Task  1.6.7) 

•  Requirements  traceability  matrix  modifications  (Source:  Task  1.6.6) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Draft  revised  V&V  Plan 

1.6.9  Circulate  the  draft  revised  V&V  Plan  for  review  and  collect  comments  and  recommendations  for 
change. 

Information  Required: 

•  Draft  revised  V&V  Plan  (Source:  Task  1.6.8) 

Information  Produced: 

•  Comments  and  recommendations  for  change  received  on  draft  revised  V&V  Plan 

1.6.10  Modify  the  draft  revised  V&V  Plan  from  the  comments  and  recommendations  for  change  received. 
Information  Required: 

•  Draft  revised  V&V  Plan  (Source:  Task  1.6.8) 
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•  Comments  and  recommendations  for  change  received  on  draft  revised  V&V  Plan 
(Source:  Task  1.6.9) 

Information  Produced: 

•  Modified  V&V  Plan 

1 .6. 1 1  Analyze  the  modified  V&V  Plan  for  internal  consistency,  correct  any  inconsistencies  and  prepare  the 
final  V&V  Plan. 

Information  Required: 

•  Modified  V&V  Plan  (Source:  Task  1.6.10) 

Information  Produced: 

•  Final  revised  V&V  Plan 

1 .6. 12  Update  the  V&V  Plan  versioning  history  associated  with  the  V&V  Plan. 

Information  Required: 

•  V&V  Plan  versioning  history  (Source:  Activity  1.5) 

Information  Produced: 

•  V&V  Plan  versioning  updates 

1 .6. 1 3  Analyze  the  V&V  Plan  execution  results,  deviations  from  the  expected  plan  results  and  modifications 
to  the  plan  to  develop  the  lessons  learned  from  exercising  the  V&V  process. 

Information  Required: 

•  V&V  Plan  execution  results  (Source:  Task  1.6.2) 

•  Deviations  from  the  expected  V&V  Plan  results  (Source:  Task  1.6.3) 

•  Existing  V&V  Plan  modifications  from  Accreditation  Plan  revisions  (Source:  Task  1.6.4) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Collected  lessons  learned  from  exercising  the  V&V  process 
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A1 .3  PHASE  2  -  APPLY  RELEVANT  HISTORICAL  INFORMATION 

Activity  2.1  -  Collect  and  Analyze  Developer  Accounts  of  M&S  Capabilities  and  Limitations 

Activity  Description:  The  M&S  Developer  possesses  unique  insight  into  its  capabilities  and  limitations,  and 
this  insight  can  contribute  greatly  to  the  knowledge  of  the  M&S’s  capabilities.  In  this  activity,  the  V&V 
practitioner  locates,  obtains,  and  analyzes  developer  accounts  of  a  M&S’s  capabilities  and  limitations.  These 
Developer’s  accounts  can  come  from  any  documentation  produced  by  the  Developer’s  activities  and  interviews 
concerning  the  development,  testing,  and  maintenance  of  the  M&S. 

Activity  Tailoring  Guidance: 

•  This  activity  may  not  be  performed  if  the  Developer  has  produced  no  documentation  from  which 
knowledge  of  the  M&S’s  capabilities  can  be  abstracted. 

•  This  activity  should  not  be  performed  if  the  Developer  cannot  produce  any  documentation  of  the 
M&S’s  configuration  management  history. 

Activity  Notes: 

•  This  activity  may  still  apply  to  new  M&S  situations  for  which  no  history  exists,  since  Developers 
often  produce  useful  documentation  during  the  development  process  (e.g.,  user’s  manuals,  training 
materials). 

•  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified  in  the 
V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  This  activity  assumes  that  the  M&S  configuration  management  history  has  not  been 
obtained  through  prior  activities.  (See  the  guidance  note  above) 

Task  Descriptions: 

2.1.1  Obtain  the  M&S  configuration  management  history  from  the  Developer. 

Information  Required: 

•  M&S  configuration  management  history  (Source:  Developer) 

Information  Produced: 

•  M&S  configuration  management  history 

2. 1 .2  Obtain  any  available  documentation  produced  by  the  Developer’s  activities. 

Information  Required: 

•  Developer’s  documentation  about  the  M&S  (Source:  Developer) 

Information  Produced: 

•  Developer’s  documentation  about  the  M&S 

2.1.3  Analyze  the  Developer’s  documentation  with  the  M&S  configuration  management  history  to  identify 
that  documentation  that  is  relevant  to  the  current  M&S  configuration. 
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Information  Required. 

•  Developer’s  documentation  about  the  M&S  (Source:  Task  2.1.2) 

•  M&S  configuration  management  history  (Source:  Task  2.1.1) 

Information  Produced: 

•  Relevant  Developer’s  documentation 

2.1.4  Analyze  the  relevant  Developer’s  documentation  with  the  acceptability  criteria  to  identify  that 
documentation  that  applies  to  the  intended  use. 

Information  Required: 

•  Developer’s  documentation  relevant  to  the  current  M&S  configuration  (Source:  Task  2. 1 .3) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

Information  Produced: 

•  Relevant  and  applicable  Developer’s  documentation 

2.1.5  Analyze  the  Relevant  and  applicable  Developer’s  documentation  to  determine  the  capabilities  and 
limitations  of  the  current  M&S  configuration. 

Information  Required: 

•  Relevant  and  applicable  Developer’s  documentation  (Source:  Task  2.1.4) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  documentation 

2.1.6  Analyze  the  M&S  capabilities  and  limitations,  the  Relevant  and  applicable  Developer’s  documentation 
and  the  acceptability  criteria  to  identify  any  gaps  and  inconsistencies  that  exist  in  the  evidence  from  the 
Developer’s  documentation. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  documentation 
(Source:  Task  2.1.5) 

•  Relevant  and  applicable  Developer’s  documentation  (Source:  Task  2.1.4) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  qualify  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Description  of  the  gaps  and  inconsistencies  in  the  evidence  from  the  Developer’s  documentation 

2. 1 .7  Interview  the  Developer,  where  needed  and  practical,  to  fill  any  gaps,  resolve  any  inconsistencies, 
describe  any  gaps  or  inconsistencies  that  cannot  be  resolved  and  document  the  results  from  these 
interviews. 
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Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  documentation 
(Source:  Task  2.1.5) 

•  Description  of  the  gaps  and  inconsistencies  in  the  evidence  from  the  Developer’s  documentation 
(Source:  Task  2.1.6) 

•  Information  from  interviews  with  the  Developer  (Source:  Developer) 

Information  Produced: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  accounts 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  Developer’s  accounts 

2.1.8  Prepare  citations  of  the  Relevant  and  applicable  Developer’s  accounts  and  resolve  any  citation 
problems. 

Information  Required: 

•  M&S  configuration  management  history  (Source:  Task  2.1.1) 

•  Relevant  and  applicable  Developer’s  documentation  (Source:  Task  2.1.4) 

•  Information  from  interviews  with  the  Developer  (Source:  Developer) 

Information  Produced: 

•  Citations  of  information  from  Developer’s  accounts 


Activity  2.2  -  Collect  and  Analyze  the  V&V  and  Testing  Histories 

Activity  Description:  Any  available  documentation  of  prior  V&V  or  testing  activities  can  contribute 
tremendously  to  an  ongoing  V&V  effort.  In  this  activity,  the  V&V  practitioner  locates,  obtains  and  analyzes  any 
available  prior  V&V  and  testing  reports.  Often  testing  activities  perform  many  of  the  same  functions  that  V&V 
does  and  can  be  leveraged  directly. 

Activity  Tailoring  Guidance: 

•  This  activity  should  not  be  performed  if  no  documentation  of  prior  VV&A  or  testing  activities  exists. 

•  This  activity  should  not  be  performed  if  the  Developer  cannot  produce  any  documentation  of  the 
M&S’s  configuration  management  history. 

•  If  the  M&S  configuration  management  history  has  been  obtained  in  a  prior  activity  then  that  task  does 
not  need  to  be  performed  in  this  activity. 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  This  activity  assumes  that  the  M&S  configuration  management  history  has  not  been 
obtained  through  prior  activities.  (See  the  guidance  note  above) 

Task  Descriptions: 

2.2. 1  Obtain  the  M&S  configuration  management  history. 
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Information  Required: 

•  M&S  configuration  management  history  (Source:  Developer) 

Information  Produced: 

•  M&S  configuration  management  history 

2.2.2  Obtain  documentation  produced  by  any  prior  VV&A  efforts. 

Information  Required: 

•  Documentation  from  prior  VV&A  efforts  (Source:  VV&A  Teams) 

Information  Produced: 

•  Documentation  from  prior  VV&A  efforts 

2.2.3  Obtain  documentation  produced  by  any  prior  testing  efforts. 

Information  Required: 

•  Documentation  from  prior  testing  efforts  (Source:  Testing  Teams) 

Information  Produced: 

•  Documentation  from  prior  testing  efforts 

2.2.4  Analyze  the  documentation  from  prior  VV&A  efforts  and  the  M&S  configuration  management  history 
to  identify  that  documentation  that  is  relevant  to  the  current  M&S  configuration. 

Information  Required: 

•  Documentation  from  prior  VV&A  efforts  (Source:  Task  2.2.2) 

•  M&S  configuration  management  history  (Source:  Task  2.2.1) 

Information  Produced: 

•  Relevant  documentation  from  prior  VV&A  efforts 

2.2.5  Analyze  the  documentation  from  prior  testing  efforts  and  the  M&S  configuration  management  history 
to  identify  that  documentation  that  is  relevant  to  the  current  M&S  configuration. 

Information  Required 

•  Documentation  from  prior  testing  efforts  (Source:  Task  2.2.3) 

•  M&S  configuration  management  history  (Source:  Task  2.2.1) 

Information  Produced: 

•  Relevant  documentation  from  prior  testing  efforts 

2.2.6  Analyze  the  relevant  documentation  from  prior  VV&A  efforts  and  the  acceptability  criteria  to  identify 
that  documentation  that  applies  to  the  intended  use. 

Information  Required: 

•  Relevant  documentation  from  prior  VV&A  efforts  (Source:  Task  2.2.4) 
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•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

Information  Produced: 

•  Relevant  and  applicable  documentation  from  prior  VV&A  efforts 

2.2.7  Analyze  the  relevant  documentation  from  prior  testing  efforts  and  the  acceptability  criteria  to  identify 
that  documentation  that  applies  to  the  intended  use. 

Information  Required: 

•  Relevant  documentation  from  prior  testing  efforts  (Source:  Task  2.2.5) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

Information  Produced: 

•  Relevant  and  applicable  documentation  from  prior  testing  efforts 

2.2.8  Analyze  the  Relevant  and  applicable  documentation  from  prior  VV&A  efforts  to  determine  the 
capabilities  and  limitations  of  the  current  M&S  configuration. 

Information  Required: 

•  Relevant  and  applicable  documentation  from  prior  VV&A  efforts  (Source:  Task  2.2.6) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  relevant  and  applicable 
documentation  of  prior  VV&A  efforts 

2.2.9  Analyze  the  Relevant  and  applicable  documentation  from  prior  testing  efforts  to  determine  the 
capabilities  and  limitations  of  the  current  M&S  configuration. 

Information  Required: 

•  Relevant  and  applicable  documentation  from  prior  testing  efforts  (Source:  Task  2.2.7) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  relevant  and  applicable 
documentation  of  prior  testing  efforts. 

2.2.10  Analyze  the  M&S  capabilities  and  limitations  from  the  relevant  and  applicable  documentation  of  prior 
VV&A  efforts  and  the  acceptability  criteria  to  identify  any  gaps  and  inconsistencies  that  may  exist  in 
the  evidence  from  that  documentation. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  relevant  and  applicable 
documentation  of  prior  VV&A  efforts  (Source:  Task  2.2.8) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 
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•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced 

•  Description  of  the  gaps  and  inconsistencies  in  the  evidence  from  the  documentation  of  prior 
VV&A  efforts 

2.2.1 1  Analyze  the  M&S  capabilities  and  limitations  from  the  relevant  and  applicable  documentation  of  prior 
testing  efforts  and  the  acceptability  criteria  to  identify  any  gaps  and  inconsistencies  that  may  exist  in 
the  evidence  from  that  documentation. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  relevant  and  applicable 
documentation  of  prior  testing  efforts  (Source:  Task  2.2.9) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Description  of  the  gaps  and  inconsistencies  in  the  evidence  from  the  documentation  of  prior 
testing  efforts 

2.2. 12  Interview  the  VV&A  teams,  where  needed  and  practical,  to  fill  any  gaps,  resolve  any  inconsistencies, 
describe  any  gaps  or  inconsistencies  that  cannot  be  resolved  and  document  the  results  from  these 
interviews. 

Information  Required: 

•  Description  of  the  gaps  and  inconsistencies  in  the  evidence  from  the  documentation  of  prior 
VV&A  efforts  (Source:  Task  2.2.10) 

•  Information  from  interviews  with  the  VV&A  teams  (Source:  VV&A  Teams) 

Information  Produced: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  VV&A  histories 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  evidence  from  the  VV&A  histories 

2.2.13  Interview  the  testing  teams,  where  needed  and  practical,  to  fill  any  gaps,  resolve  any  inconsistencies, 
describe  any  gaps  or  inconsistencies  that  cannot  be  resolved  and  document  the  results  from  these 
interviews. 

Information  Required: 

•  Description  of  the  gaps  and  inconsistencies  in  the  evidence  from  the  documentation  of  prior 
testing  efforts  (Source:  Task  2.2. 11) 

•  Information  from  interviews  with  the  testing  teams  (Source:  Testing  Teams) 

Information  Produced: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  testing  histories 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  evidence  from  the  testing  histories 
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2.2. 14  Assemble  the  information  on  the  M&S  capabilities  and  limitations  from  prior  VV&A  and  testing  efforts 
into  a  consistent  description. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  VV&A  histories 
(Source:  Task  2.2. 12) 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  testing  histories 
(Source:  Task  2.2. 13) 

Information  Produced: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  prior  VV&A  and  testing 

2.2.15  Assemble  the  information  on  the  unresolved  gaps  and  inconsistencies  from  prior  VV&A  and  testing 
efforts  into  a  consistent  description. 

Information  Required: 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  evidence  from  the  VV&A  histories 
(Source:  Task  2.2. 12) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  evidence  from  the  testing  histories 
(Source:  Task  2.2.13) 

Information  Produced: 

•  Integrated  description  of  the  unresolved  gaps  and  limitations  the  VV&A  and  testing  histories 

2.2.16  Prepare  citations  of  the  relevant  and  applicable  VV&A  and  testing  histories  and  resolve  any  citation 
problems. 

Information  Required: 

•  M&S  configuration  management  history  (Source:  Task  2.2.1) 

•  Relevant  and  applicable  documentation  from  prior  VV&A  activities  (Source:  Task  2.2.4) 

•  Relevant  and  applicable  documentation  from  prior  testing  activities  (Source:  Task  2.2.5) 

•  Information  from  interviews  with  the  VV&A  teams  (Source:  VV&A  Teams) 

•  Information  from  interviews  with  the  testing  teams  (Source:  Testing  Teams) 

Information  Produced: 

•  Citations  of  information  from  the  VV&A  and  testing  histories 


Activity  2.3  -  Collect  and  Analyze  the  Prior  Use  History 

Activity  Description:  Like  the  information  produced  by  prior  V&V  and  testing  activities,  documentation  of 
prior  uses  of  a  M&S  can  provide  information  about  the  M&S’s  validity.  In  this  activity,  the  V&V  practitioner 
locates,  obtains  and  analyzes  any  documentation  of  prior  use  history.  The  products  of  prior  M&S  use  can  be 
reports  or  technical  papers  that  document  the  results  from  studies,  training  exercises,  experiments,  and  design 
activities. 

Activity  Tailoring  Guidance: 

•  This  activity  should  not  be  performed  if  no  documentation  of  prior  use  exists. 
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•  This  activity  should  not  be  performed  if  the  Developer  cannot  produce  any  documentation  of  the 
M&S’s  configuration  management  history. 

•  If  the  M&S  configuration  management  history  has  been  obtained  in  a  prior  activity  then  that  task  does 
not  need  to  be  performed  in  this  activity. 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  This  activity  assumes  that  the  M&S  configuration  management  history  has  not  been 
obtained  through  prior  activities.  (See  the  guidance  note  above) 

Task  Descriptions: 

2.3.1  Obtain  the  M&S  configuration  management  history. 

Information  Required: 

•  M&S  configuration  management  history  (Source:  Developer) 

Information  Produced: 

•  M&S  configuration  management  history 

2.3.2  Obtain  any  documentation  produced  by  prior  uses  of  the  M&S. 

Information  Required: 

•  Documentation  from  prior  uses  of  the  M&S  (Source:  Users) 

Information  Produced: 

•  Documentation  from  prior  uses  of  the  M&S 

2.3.3  Analyze  the  documentation  from  prior  uses  and  the  M&S  configuration  management  history  to  identify 
the  documentation  that  is  relevant  to  the  current  M&S  configuration. 

Information  Required: 

•  Documentation  from  prior  uses  of  the  M&S  (Source:  Task  2.3.2) 

•  M&S  configuration  management  history  (Source:  Task  2.3.1) 

Information  Produced: 

•  Relevant  documentation  from  prior  M&S  uses 

2.3.4  Analyze  the  relevant  documentation  from  prior  uses  and  the  acceptability  criteria  to  identify  the 
documentation  that  applies  to  the  intended  use. 

Information  Required: 

•  Relevant  documentation  from  prior  M&S  uses  (Source:  Task  2.3.3) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

Information  Produced: 

•  Relevant  and  applicable  documentation  from  prior  uses 
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2.3.5  Analyze  the  relevant  and  applicable  documentation  from  prior  uses  to  determine  the  capabilities  and 
limitations  of  the  current  M&S  configuration. 

Information  Required: 

•  Relevant  and  applicable  documentation  from  prior  uses  (Source:  Task  2.3.4) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  relevant  and  applicable 
documentation  from  prior  uses 

2.3.6  Analyze  the  M&S  capabilities  and  limitations  from  the  relevant  and  applicable  documentation  from 
prior  uses  and  the  acceptability  criteria  to  identify  any  gaps  and  inconsistencies  that  may  exist  in  the 
evidence  from  that  documentation. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Relevant  and  applicable 
documentation  from  prior  uses  (Source:  Task  2.3.5) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Description  of  the  gaps  and  inconsistencies  in  the  evidence  from  the  documentation  of  prior  uses 

2.3.7  Interview  the  users,  where  needed  and  practical,  to  fill  any  gaps,  resolve  any  inconsistencies,  describe 
any  gaps  or  inconsistencies  that  cannot  be  resolved  and  document  the  results  of  these  interviews. 

Information  Required: 

•  Description  of  the  gaps  and  inconsistencies  in  the  evidence  from  the  documentation  of  prior  uses 
(Source:  Task  2.3.6) 

•  Information  from  interviews  with  the  users  (Source:  Users) 

Information  Produced: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  use  history 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  evidence  from  the  use  history 

2.3.8  Prepare  citations  of  the  relevant  and  applicable  use  history  and  resolve  any  citation  problems. 
Information  Required: 

•  M&S  configuration  management  history  (Source:  Developer) 

•  Relevant  and  applicable  documentation  from  prior  uses  (Source:  Task  2.3.4) 

•  Information  from  interviews  with  the  users  (Source:  Users) 

Information  Produced: 

•  Citations  of  information  from  the  use  history 
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Activity  2.4  -  Integrate  and  Employ  the  Historical  Evidence 

Activity  Description:  In  this  activity,  the  V&V  practitioner  integrates  any  relevant  historical  evidence  into  a 
coherent  view  of  an  M&S’s  capabilities  and  limitations. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

2.4. 1  Analyze  the  historical  evidence  produced  by  the  prior  activities  to  generate  an  integrated  description 
of  the  M&S  capabilities  and  limitations. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  accounts 
(Source:  Activity  2.1) 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  VV&A  and  testing  histories 
(Source:  Activity  2.2) 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  use  history  (Source:  Activity  2.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  historical  evidence 

2.4.2  Analyze  the  gaps  and  inconsistencies  identified  by  the  prior  activities  using  the  integrated  description  of 
the  M&S  capabilities  and  limitations  from  the  historical  evidence  to  reconcile  them  and  produce  a 
description  of  the  unresolved  gaps  and  inconsistencies  in  the  historical  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  historical  evidence 
(Source:  Task  2.4.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  evidence  from  the  Developer’s 
accounts  (Source:  Activity  2.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  evidence  from  the  VV&A  and 
testing  histories  (Source:  Activity  2.2) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  evidence  from  the  use  history 
(Source:  Activity  2.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  historical  evidence 

2.4.3  Analyze  the  integrated  description  of  M&S  capabilities  and  limitations,  description  of  the  unresolved 
gaps  and  inconsistencies  in  the  integrated  historical  evidence,  and  acceptability  criteria  to  estimate  the 
coverage  of  the  integrated  historical  evidence. 
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Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  historical  evidence 
(Source:  Task  2.4.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  historical  evidence 
(Source:  Task  2.4.2) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  coverage  of  the  historical  evidence 

2.4.4  Analyze  the  integrated  description  of  M&S  capabilities  and  limitations,  description  of  the  unresolved 
gaps  and  inconsistencies  in  the  integrated  historical  evidence,  and  estimated  coverage  in  the  historical 
evidence  to  estimate  the  uncertainties  in  the  integrated  historical  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  historical  evidence 
(Source:  Task  2.4.1) 

•  Estimated  coverage  of  the  historical  evidence  (Source:  Task  2.4.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  uncertainties  in  the  historical  evidence 

2.4.5  Analyze  the  acceptability  criteria,  the  integrated  description  of  M&S  capabilities  and  limitations  from 
the  historical  evidence  and  the  estimated  uncertainties  in  the  historical  evidence  to  identify  the  factors 
that  may  constrain  use. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  historical  evidence 
(Source:  Task  2.4.1) 

•  Estimated  uncertainties  in  the  historical  evidence  (Source:  Task  2.4.4) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Factors  that  may  constrain  use  implied  by  the  historical  evidence 

2.4.6  Analyze  the  integrated  description  of  the  M&S  capabilities  and  limitations  from  the  historical  evidence, 
the  unresolved  gaps  and  inconsistencies  in  the  historical  evidence,  the  estimated  coverage  of  uncertainties 
in  the  historical  evidence  and  the  acceptability  criteria  to  identify  candidate  output  sampling  areas. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  historical  evidence 
(Source:  Task  2.4.1) 
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•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  historical  evidence 
(Source:  Task  2.4.2) 

•  Estimated  coverage  of  the  historical  evidence  (Source:  Task  2.4.3) 

•  Estimated  uncertainties  in  the  historical  evidence  (Source:  Task  2.4.4) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Candidate  output  sampling  areas  suggested  by  the  historical  evidence 

2.4.7  Update  the  requirements  traceability  matrix  with  the  historical  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  historical  evidence 
(Source:  Task  2.4.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  historical  evidence 
(Source:  Task  2.4.2) 

•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

2.4.8  Prepare,  from  the  individual  citation  lists,  an  integrated  list  of  citations  of  relevant  and  applicable 
information  and  resolve  any  citation  problems. 

Information  Required: 

•  Citations  of  information  from  Developer’s  accounts  (Source:  Activity  2.1) 

•  Citations  of  information  from  the  VV&A  and  testing  histories  (Source:  Activity  2.2) 

•  Citations  of  information  from  the  use  history  (Source:  Activity  2.3) 

Information  Produced: 

•  Integrated  citations  supporting  the  compilation  and  analysis  of  the  historical  evidence 
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A1.4  PHASE  3  -  VERIFY  AND  VALIDATE  THE  M&S  CONCEPTUAL  MODEL 

Activity  3.1  -  Verify  the  Conceptual  Model 

Activity  Description:  This  verifies  the  M&S  conceptual  model. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes: 

•  Analysis  to  discover  computational  anomalies  is  constrained  to  the  conceptual  model  (specifically, 
the  mathematical  model)  because  that  seems  to  be  the  only  artifact  that  lends  itself  to  analysis. 

•  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified  in  the 
V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  An  M&S  conceptual  model  exists  and  is  available  to  the  V&V  effort. 

Task  Descriptions: 

3.1.1  Obtain  the  M&S  conceptual  model. 

Information  Required: 

•  M&S  conceptual  model  (Source:  Developer) 

Information  Produced: 

•  M&S  conceptual  model 

3.1.2  Analyze  the  M&S  conceptual  model  for  the  possible  existence  of  computational  anomalies. 
Information  Required: 

•  M&S  conceptual  model  (Source:  Task  3.1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Computational  anomalies  discovered  in  the  M&S  conceptual  model 

3.1.3  Analyze  the  M&S  conceptual  model  to  determine  its  internal  consistency. 

Information  Required: 

•  M&S  conceptual  model  (Source:  Task  3.1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  within  the  M&S  conceptual  model 

3.1.4  Analyze  the  M&S  requirements  and  the  M&S  conceptual  model  to  determine  the  consistency  between 
them. 
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Information  Required: 

•  M&S  requirements  (Source:  Activity  1.1) 

•  M&S  conceptual  model  (Source:  Task  3.1.1) 

Information  Produced: 

•  Inconsistencies  discovered  between  the  M&S  requirements  and  the  M&S  conceptual  model. 

3.1.5  Analyze  the  M&S  conceptual  model  to  verify  the  input  functionality  that  it  represents. 

Information  Required: 

•  M&S  conceptual  model  (Source:  Task  3.1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  in  the  input  design  represented  in  the  M&S  conceptual  model 

3.1.6  Analyze  the  M&S  conceptual  model  to  verily  the  output  functionality  that  it  represents. 

Information  Required: 

•  M&S  conceptual  model  (Source:  Task  3.1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  in  the  output  design  represented  in  the  M&S  conceptual  model 

3.1.7  Analyze  the  computational  anomalies  and  the  inconsistencies  discovered  during  the  M&S  conceptual 
model  verification  to  develop  an  integrated  description  of  the  inconsistencies  discovered  in  the  M&S 
conceptual  model. 

Information  Required: 

•  Computational  anomalies  discovered  in  the  M&S  conceptual  model  (Source:  Task  3.1.2) 

•  Inconsistencies  discovered  within  the  M&S  conceptual  model  (Source:  Task  3.1.3) 

•  Inconsistencies  discovered  between  the  M&S  requirements  and  the  M&S  conceptual  model 
(Source:  Task  3.1.4) 

•  Inconsistencies  discovered  in  the  input  design  represented  in  the  M&S  conceptual  model 
(Source:  Task  3.1.5) 

•  Inconsistencies  discovered  in  the  output  design  represented  in  the  M&S  conceptual  model 
(Source:  Task  3.1.6) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  conceptual  model 

•  Verified  M&S  conceptual  model 


Activity  3.2  -  Validate  the  Conceptual  Model 

Activity  Description:  This  activity  validates  the  M&S  conceptual  model. 
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Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  This  activity  assumes  that  the  M&S  conceptual  model  has  been  obtained  and  verified 
as  part  of  previous  activities. 

Task  Descriptions: 

3.2. 1  Analyze  the  verified  M&S  conceptual  model  and  the  acceptability  criteria  defining  needed  functionality 
to  determine  the  functionality  needs  that  are  met;  identify  those  criteria  that  are  not  met;  and  identify 
those  criteria  for  which  insufficient  information  is  available. 

Information  Required: 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  Verified  M&S  conceptual  model  (Source:  Activity  3.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Acceptability  criteria  defining  needed 

•  Acceptability  criteria  defining  needed 
meet 

•  Acceptability  criteria  defining  needed 
from  the  M&S  conceptual  model 

3.2.2  Analyze  the  verified  M&S  conceptual  model,  the  acceptability  criteria  defining  needed  quality  and  the 
validation  referent  to  determine  the  quality  needs  that  are  met,  identify  those  criteria  that  are  not  met 
and  identify  those  criteria  for  which  insufficient  information  is  available. 

Information  Required: 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  Verified  M&S  conceptual  model  (Source:  Activity  3.1) 

•  Validation  Referent  (Source:  Preceding  accreditation  activity) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Acceptability  criteria  defining  needed  quality  that  the  M&S  conceptual  model  meets 

•  Acceptability  criteria  defining  needed  quality  that  the  M&S  conceptual  model  does  not  meet 

•  Acceptability  criteria  defining  needed  quality  for  which  insufficient  information  is  available  from 

the  M&S  conceptual  model 

3.2.3  Analyze  the  acceptability  criteria  that  the  verified  M&S  conceptual  model  meets  and  the  verified 
M&S  conceptual  model  to  determine  the  M&S  capabilities  that  the  valid  M&S  conceptual  model 
supports. 


functionality  that  the  M&S  conceptual  model  meets 
functionality  that  the  M&S  conceptual  model  does  not 

functionality  for  which  insufficient  information  is  available 
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Information  Required: 

•  Acceptability  criteria  defining  needed  functionality  that  the  M&S  conceptual  model  meets 
(Source:  Task  3.2.1) 

•  Acceptability  criteria  defining  needed  quality  that  the  M&S  conceptual  model  meets 
(Source:  Task  3.2.2) 

•  Verified  M&S  conceptual  model  (Source:  Activity  3.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  capabilities  that  the  valid  M&S  conceptual  model  can  support 

3.2.4  Analyze  the  acceptability  criteria  that  the  verified  M&S  conceptual  model  does  not  meet  and  the 
verified  M&S  conceptual  model  to  determine  the  M&S  limitations  implied  by  the  valid  M&S  conceptual 
model. 

Information  Required: 

•  Acceptability  criteria  defining  needed  functionality  that  the  M&S  conceptual  model  does  not 
meet  (Source:  Task  3.2.1) 

•  Acceptability  criteria  defining  needed  quality  that  the  M&S  conceptual  model  does  not  meet 
(Source:  Task  3.2.2) 

•  Verified  M&S  conceptual  model  (Source:  Activity  3.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  limitations  implied  by  the  M&S  conceptual  model  validation 

•  Verified  and  validated  M&S  conceptual  model 

3.2.5  Analyze  the  acceptability  criteria  for  which  insufficient  information  is  available  and  the  verified  M&S 
conceptual  model  to  determine  the  information  gaps  implied  by  that  model. 

Information  Required: 

•  Acceptability  criteria  defining  needed  functionality  for  which  insufficient  information  is  available 
from  the  M&S  conceptual  model  (Source:  Task  3.2.1) 

•  Acceptability  criteria  defining  needed  quality  for  which  insufficient  information  is  available  from 
the  M&S  conceptual  model  (Source:  Task  3.2.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Information  gaps  in  the  conceptual  model  validation  evidence 

Activity  3.3  -  Verify  and  Validate  the  Use  Scenarios 

Activity  Description:  This  activity  verifies  and  validates  the  use  scenarios. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 
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Activity  Assumptions:  An  M&S  conceptual  model  exists  that  has  been  verified  and  validated. 

Task  Descriptions: 

3.3.1  Obtain  the  scenarios  for  the  intended  use. 

Information  Required: 

•  User  Scenarios  (Source:  User) 

Information  Produced: 

•  Collected  scenarios 

3.3.2  Analyze  the  individual  collected  scenarios  to  determine  their  internal  consistency. 

Information  Required: 

•  Collected  scenarios  (Source:  Task  3.3.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  within  the  individual  collected  scenarios 

3.3.3  Analyze  the  collected  scenarios  to  determine  the  consistency  between  them. 

Information  Required: 

•  Collected  scenarios  (Source:  Task  3.3.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  between  the  collected  scenarios 

3.3.4  Analyze  the  collected  scenarios  to  determine  their  consistency  with  the  intended  use. 
Information  Required: 

•  Statement  of  intended  use  (Source:  Preceding  accreditation  activity) 

•  Collected  scenarios  (Source:  Task  3.3.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  between  the  collected  scenarios  and  the  intended  use 

3.3.5  Analyze  the  inconsistencies  discovered  during  the  scenario  verification  to  develop  an  integrated 
description  of  the  scenario  inconsistencies  and  the  verified  scenarios. 

Information  Required: 

•  Inconsistencies  within  the  individual  collected  scenarios  (Source:  Task  3.3.2) 

•  Inconsistencies  between  the  collected  scenarios  (Source:  Task  3.3.3) 

•  Inconsistencies  between  the  collected  scenarios  and  the  intended  use  (Source:  Task  3.3.4) 

•  V&V  Plan  (Source:  Activity  1.6) 
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Information  Produced: 

•  Integrated  description  of  the  scenario  inconsistencies 

•  Verified  collected  scenarios 

3.3.6  Analyze  the  verified  collected  scenarios,  scenario  inconsistencies  and  the  acceptability  criteria  defining 
needed  functionality  to  determine  the  functionality  needs  that  are  met,  identify  those  criteria  that  are  not 
met  and  identify  those  criteria  for  which  insufficient  information  is  available. 

Information  Required: 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Integrated  description  of  the  scenario  inconsistencies  (Source:  Task  3.3.5) 

•  Verified  collected  scenarios  (Source:  Task  3.3.5) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Acceptability  criteria  defining  needed  functionality  that  the  scenarios  meet 

•  Acceptability  criteria  defining  needed  functionality  that  the  scenarios  do  not  meet 

•  Acceptability  criteria  defining  needed  functionality  for  which  insufficient  information  is  available 
from  the  scenarios 

3.3.7  Analyze  the  verified  collected  scenarios,  scenario  inconsistencies,  the  acceptability  criteria  defining 
needed  quality  and  the  validation  referent  to  determine  the  quality  needs  that  are  met,  identify  those 
criteria  that  are  not  met  and  identify  those  criteria  for  which  insufficient. 

Information  Required: 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  Integrated  description  of  the  scenario  inconsistencies  (Source:  Task  3.3.5) 

•  Verified  collected  scenarios  (Source:  Task  3.3.5) 

•  Validation  Referent  (Source:  Preceding  accreditation  activity) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Acceptability  criteria  defining  needed  quality  that  the  scenarios  meet 

•  Acceptability  criteria  defining  needed  quality  that  the  scenarios  do  not  meet 

•  Acceptability  criteria  defining  needed  quality  for  which  insufficient  information  is  available  from 
the  scenarios 

3.3.8  Analyze  the  acceptability  criteria  that  the  verified  collected  scenarios  meet  and  the  M&S  conceptual 
model  to  determine  the  M&S  capabilities  that  valid  scenarios  can  support. 

Information  Required: 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  Acceptability  criteria  defining  needed  functionality  that  the  scenarios  meet  (Source:  Task  3.3.6) 

•  Acceptability  criteria  defining  needed  quality  that  the  scenarios  meet  (Source:  Task  3.3.7) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  capabilities  that  the  valid  scenarios  can  support 
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3.3.9  Analyze  the  acceptability  criteria  that  the  verified  collected  scenarios  do  not  meet  and  the  M&S 
conceptual  model  to  determine  the  M&S  limitations  implied  by  the  scenarios. 

Information  Required: 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  Acceptability  criteria  defining  needed  functionality  that  the  scenarios  do  not  meet 
(Source:  Task  3.3.6) 

•  Acceptability  criteria  defining  needed  quality  that  the  scenarios  do  not  meet  (Source:  Task  3.3.7) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  limitations  implied  by  the  scenario  validation 

3.3.10  Analyze  the  acceptability  criteria  for  which  insufficient  information  is  available  from  the  scenario 
validation  and  the  M&S  conceptual  model  to  determine  the  information  gaps  implied  by  those  sets. 

Information  Required: 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  Acceptability  criteria  defining  needed  functionality  for  which  insufficient  information  is  available 
from  the  scenarios  (Source:  Task  3.3.6) 

•  Acceptability  criteria  defining  needed  quality  for  which  insufficient  information  is  available  from 
the  scenarios  (Source:  Task  3.3.7) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Information  gaps  implied  by  the  scenario  validation 

3.3.11  Assemble  and  communicate  the  results  of  the  scenario  verification  and  validation. 

Information  Required: 

•  Integrated  description  of  the  scenario  inconsistencies  (Source:  Task  3.3.5) 

•  M&S  capabilities  that  the  valid  scenarios  can  support  (Source:  Task  3.3.8) 

•  M&S  limitations  implied  by  the  scenario  validation  (Source:  Task  3.3.9) 

•  Information  gaps  implied  by  the  scenario  validation  (Source:  Task  3.3.10) 

Information  Produced: 

•  Integrated  description  of  scenario  verification  and  validation  results 

•  Verified  and  validated  collected  scenarios 

Activity  3.4  -  Integrate  and  Employ  the  Conceptual  Model  V&V  Evidence 

Activity  Description:  This  integrates  the  scenario  and  M&S  conceptual  model  verification  and  validation 
results. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 
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Activity  Assumptions:  (Blank) 

Task  Descriptions: 

3.4.1  Analyze  1)  the  scenarios  and  M&S  conceptual  model,  2)  the  descriptions  of  the  scenario  and  M&S 
conceptual  model  inconsistencies,  and  3)  the  M&S  capabilities  that  the  M&S  conceptual  model  and 
scenarios  support  to  produce  an  integrated  description  of  the  M&S  capabilities  supported  by  both  the 
M&S  conceptual  model  and  scenarios. 

Information  Required: 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  Verified  and  validated  collected  scenarios  (Source:  Activity  3.3) 

•  Integrated  description  of  the  M&S  conceptual  model  inconsistencies  (Source:  Activity  3.1) 

•  Integrated  description  of  the  scenario  inconsistencies  (Source:  Activity  3.3) 

•  M&S  capabilities  that  the  valid  M&S  conceptual  model  supports  (Source:  Activity  3.2) 

•  M&S  capabilities  that  the  valid  scenarios  supports  (Source:  Activity  3.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  M&S  capabilities  that  the  scenarios  and  M&S  conceptual  model  can 
support 

3.4.2  Analyze  the  scenarios  and  M&S  conceptual  model,  the  integrated  descriptions  of  the  scenario  and  M&S 
conceptual  model  inconsistencies,  the  M&S  limitations  implied  by  the  scenario  and  M&S  conceptual 
model  validation  to  produce  an  integrated  description  of  the  M&S  limitations  implied  the  scenarios  and 
M&S  conceptual  model. 

Information  Required: 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  Verified  and  validated  collected  scenarios  (Source:  Activity  3.3) 

•  Integrated  description  of  the  M&S  conceptual  model  inconsistencies  (Source:  Activity  3.1) 

•  Integrated  description  of  the  scenario  inconsistencies  (Source:  Activity  3.3) 

•  M&S  limitations  implied  by  the  M&S  conceptual  model  validation  (Source:  Activity  3.2) 

•  M&S  limitations  implied  by  the  scenario  validation  (Source:  Activity  3.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  M&S  limitations  implied  by  the  scenario  and  M&S  conceptual 
model  V&V 

3.4.3  Analyze  the  integrated  description  of  the  M&S  capabilities  that  the  conceptual  model  can  support  and 
the  integrated  description  of  the  M&S  limitations  implied  by  the  scenario  and  the  conceptual  model 
V&V  to  produce  an  integrated  description  of  the  M&S  capabilities  and  limitations  implied  by  the 
scenario  and  conceptual  model  V&V. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  that  the  scenarios  and  M&S  conceptual  model  can 
support  (Source:  Task  3.4.1) 
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•  Integrated  description  of  the  M&S  limitations  implied  by  the  scenario  and  M&S  conceptual 
model  V&V  (Source:  Task  3.4.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  implied  by  the  scenario  and  M&S 
conceptual  model  V&V 

3.4.4  Analyze  the  scenarios  and  M&S  conceptual  model,  the  acceptability  criteria,  the  integrated  descriptions 
of  the  scenario  and  M&S  conceptual  model  inconsistencies,  the  information  gaps  implied  by  the 
scenario  and  M&S  conceptual  model  validation  to  produce  an  integrated  description  of  the  information 
gaps  and  inconsistencies  in  the  evidence  from  the  scenarios  and  the  M&S  conceptual  model. 

Information  Required: 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  Verified  and  validated  scenarios  (Source:  Activity  3.3) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  Integrated  description  of  the  M&S  conceptual  model  inconsistencies  (Source:  Activity  3.1) 

•  Integrated  description  of  the  scenario  inconsistencies  (Source:  Activity  3.3) 

•  Information  gaps  implied  by  the  M&S  conceptual  model  validation  (Source:  Activity  3.2) 

•  Information  gaps  implied  by  the  scenario  validation  (Source:  Activity  3.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  information  gaps  and  inconsistencies  in  the  evidence  from  the 
scenario  and  M&S  conceptual  model  V&V 

3.4.5  Analyze  the  acceptability  criteria,  integrated  description  of  the  M&S  capabilities  that  the  scenarios 
and  M&S  conceptual  model  can  support,  the  integrated  description  of  the  limitations  implied  by  the 
scenarios  and  M&S  conceptual  model  and  the  information  gaps  implied  by  the  scenario  and  M&S 
conceptual  model  validation  to  estimate  the  coverage  of  the  scenario  and  M&S  conceptual  model 
V&V  evidence. 

Information  Required: 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  implied  by  the  scenario  and  M&S 
conceptual  model  V&V  (Source:  Task  3.4.3) 

•  Integrated  description  of  the  information  gaps  and  inconsistencies  in  the  evidence  from  the 
scenario  and  M&S  conceptual  model  V&V  (Source:  Task  3.4.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  coverage  of  the  scenario  and  M&S  conceptual  model  V&V  evidence 

3.4.6  Analyze  the  integrated  descriptions  of  the  M&S  capabilities,  limitations  and  information  gaps  implied 
by  the  scenarios  and  M&S  conceptual  model  V&V  evidence  and  the  estimated  coverage  of  the  scenario 
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and  M&S  conceptual  model  V&V  evidence  to  estimate  the  uncertainties  in  the  scenario  and  M&S 
conceptual  model  V&V  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  implied  by  the  scenario  and  M&S 
conceptual  model  V&V  (Source:  Task  3.4.3) 

•  Integrated  description  of  the  information  gaps  and  inconsistencies  in  the  evidence  from  the 
scenario  and  M&S  conceptual  model  V&V  (Source:  Task  3.4.4) 

•  Estimated  coverage  of  the  scenario  and  M&S  conceptual  model  V&V  evidence  (Source:  Task 
3.4.5) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  uncertainties  in  the  M&S  conceptual  model  V&V  evidence 

3.4.7  Analyze  the  integrated  description  of  the  M&S  limitations  implied  by  the  scenario  and  M&S  conceptual 
model  V&V,  the  integrated  description  of  the  information  gaps  implied  by  the  scenario  and  M&S 
conceptual  model  V&V,  and  the  acceptability  criteria  to  identify  the  factors  that  may  constrain  use. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  implied  by  the  scenario  and  M&S 
conceptual  model  V&V  (Source:  Task  3.4.3) 

•  Integrated  description  of  the  information  gaps  and  inconsistencies  in  the  evidence  from  the 
scenario  and  M&S  conceptual  model  V&V  (Source:  Task  3.4.4) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Factors  that  may  constrain  use  implied  by  the  scenario  and  M&S  conceptual  model  V&V 

3.4.8  Analyze  the  integrated  description  of  the  M&S  limitations  implied  by  the  scenario  and  M&S 
conceptual  model  V&V,  the  integrated  description  of  the  information  gaps  implied  by  the  scenario  and 
M&S  conceptual  model  V&V  and  the  acceptability  criteria  to  identify  the  candidate  output  sampling 
areas  implied  by  the  scenario  and  M&S  conceptual  model  V&V. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  implied  by  the  scenario  and  M&S 
conceptual  model  V&V  (Source:  Task  3.4.3) 

•  Integrated  description  of  the  information  gaps  and  inconsistencies  in  the  evidence  from  the 
scenario  and  M&S  conceptual  model  V&V  (Source:  Task  3.4.4) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Candidate  output  sampling  areas  implied  by  the  scenario  and  M&S  conceptual  model  V&V 
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3.4.9  Update  the  requirements  traceability  matrix  with  the  evidence  from  the  M&S  conceptual  model 
verification  and  validation. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  implied  by  the  scenario  and  M&S 
conceptual  model  V&V  (Source:  Task  3.4.3) 

•  Integrated  description  of  the  information  gaps  and  inconsistencies  in  the  evidence  from  the 
scenario  and  M&S  conceptual  model  V&V  (Source:  Task  3.4.4) 

•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Requirements  traceability  matrix  updated  with  the  M&S  conceptual  model  V&V  evidence 

3.4.10  Prepare  citations  of  the  M&S  conceptual  model  and  collected  scenarios  and  resolve  any  citation 
problems. 

Information  Required: 

•  M&S  conceptual  model  (Source:  Activity  3.1) 

•  Collected  scenarios  (Source:  Activity  3.3) 

Information  Produced: 

•  Integrated  citations  supporting  the  compilation  and  analysis  of  the  M&S  conceptual  model  and 
collected  scenarios 
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A1.5  PHASE  4  -  PERFORM  SUPPLEMENTAL  VERIFICATION 
Activity  4.1  -  Determine  the  Scope  of  Supplemental  Verification  Needed 

Activity  Description:  This  activity  determines  the  scope  of  the  verification  that  will  be  done  by  the  V&V 
Team  to  supplement  that  done  by  the  Developer. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions: 

•  The  Developer  has  produced  documentation  of  the  verification  activities  that  they  have  performed. 

•  The  Developer  has  an  up-to-date  M&S  configuration  management  history. 

Task  Descriptions: 

4.1.1  Obtain  the  M&S  configuration  management  history. 

Information  Required: 

•  M&S  configuration  management  history  (Source:  Developer) 

Information  Produced: 

•  M&S  configuration  management  history 

4. 1 .2  Obtain  the  Developer’s  verification  products. 

Information  Required: 

•  Developer’s  verification  products  (Source:  Developer) 

Information  Produced: 

•  Developer’s  verification  products 

4. 1 .3  Analyze  the  Developer’s  verification  products  with  the  M&S  configuration  management  history  to 
identify  those  products  that  are  relevant  to  the  current  M&S  configuration. 

Information  Required: 

•  Developer’s  verification  products  (Source:  Task  4. 1 .2) 

•  M&S  configuration  management  history  (Source:  Task  4.1.1) 

Information  Produced: 

•  Relevant  Developer’s  verification  products 

4. 1 .4  Analyze  the  relevant  Developer’s  verification  products,  verified  and  validated  M&S  conceptual  model 
and  acceptability  criteria  to  identify  those  products  that  apply  to  the  intended  use. 
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Information  Required: 

•  Relevant  Developer’s  verification  products  (Source:  Task  4.1.3) 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

Information  Produced: 

•  Relevant  and  applicable  Developer’s  verification  products 

4. 1 .5  Analyze  the  relevant  and  applicable  Developer’s  verification  products,  verified  and  validated  M&S 
conceptual  model  and  acceptability  criteria  to  determine  the  M&S  capabilities  and  limitations  implied 
by  the  Developer’s  verification  products. 

Information  Required: 

•  Relevant  and  applicable  Developer’s  verification  products  (Source:  Task  4.1.4) 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  capabilities  and  limitations  implied  by  the  Developer’s  verification  products 

4. 1 .6  Analyze  the  relevant  and  applicable  Developer’s  verification  products,  acceptability  criteria  and  the 
M&S  capabilities  and  limitations  implied  by  the  Developer’s  verification  products  to  determine  the 
gaps  and  inconsistencies  in  the  evidence  from  the  Developer’s  verification  products. 

Information  Required: 

•  Relevant  and  applicable  Developer’s  verification  products  (Source:  Task  4.1.4) 

•  M&S  capabilities  and  limitations  implied  by  the  Developer’s  verification  products 
(Source:  Task  4.1.5) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Gaps  and  inconsistencies  in  the  evidence  from  the  Developer’s  verification  products 

4. 1 .7  Analyze  the  M&S  capabilities  and  limitations  implied  by  the  Developer’s  verification  products, 
gaps  and  inconsistencies  in  the  Developer’s  verification  products  and  acceptability  criteria  to  estimate 
the  coverage  of  the  Developer’s  verification  products. 

Information  Required: 

•  M&S  capabilities  and  limitations  implied  by  the  Developer’s  verification  products 
(Source:  Task  4.1.5) 

•  Gaps  and  inconsistencies  in  the  evidence  from  the  Developer’s  verification  products 
(Source:  Task  4.1.6) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 
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Information  Produced: 

•  Estimated  coverage  of  the  Developer’s  verification  products 

4.1.8  Analyze  the  gaps  and  inconsistencies  in  the  Developer’s  verification  products  and  estimated  coverage 
of  the  Developer’s  verification  products  and  the  V&V  Plan  to  determine  the  supplemental  verification 
needs. 

Information  Required: 

•  Estimated  coverage  of  the  Developer’s  verification  products  (Source:  Task  4. 1 .7) 

•  Gaps  and  inconsistencies  in  the  evidence  from  the  Developer’s  verification  products 
(Source:  Task  4.1.6) 

•  V&V  Plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Supplemental  verification  needs 


Activity  4.2  -  Verify  the  M&S  Design  Products 
Activity  Description:  (Blank) 

Activity  Tailoring  Guidance:  This  activity  need  not  be  performed  if  the  supplemental  verification  needs  do 
not  include  any  needs  for  verifying  the  M&S  design  products  (e.g.,  the  developer  has  already  performed  adequate 
verification  of  the  M&S  design  products). 

Activity  Notes: 

•  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified  in  the 
V&V  Plan  to  perform  the  analyses. 

•  All  design  products  can  only  be  analyzed  for  inconsistencies  and  cannot  be  tested  because  they  cannot 
be  executed  (i.e.,  only  the  M&S  executable  can  be  executed). 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

4.2.1  Analyze  the  supplemental  verification  needs  to  determine  the  needs  for  verifying  the  M&S  design 
products. 

Information  Required: 

•  Supplemental  verification  needs  (Source:  Activity  4.1) 

Information  Produced: 

•  M&S  design  product  verification  needs 

4.2.2  Obtain  the  M&S  design  products  from  the  Developer. 

Information  Required: 

•  M&S  design  products  (Source:  Developer) 
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Information  Produced: 

•  M&S  design  products 

4.2.3  Analyze  the  M&S  design  products  for  the  possible  existence  of  computational  anomalies. 
Information  Required: 

•  M&S  design  product  verification  needs  (Source:  Task  4.2.1) 

•  M&S  design  products  (Source:  Task  4.2.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Computational  anomalies  discovered  in  the  M&S  design  products 

4.2.4  Analyze  the  M&S  design  products  to  determine  their  internal  consistency. 

Information  Required: 

•  M&S  design  product  verification  needs  (Source:  Task  4.2.1) 

•  M&S  design  products  (Source:  Task  4.2.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  within  the  M&S  design  products 

4.2.5  Analyze  the  M&S  design  products  to  determine  the  consistency  between  them. 
Information  Required: 

•  M&S  design  product  verification  needs  (Source:  Task  4.2.1) 

•  M&S  design  products  (Source:  Task  4.2.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  between  the  M&S  design  products 

•  Verified  M&S  design  product 

4.2.6  Analyze  the  M&S  design  products  to  verify  the  input  functionality. 

Information  Required: 

•  M&S  design  product  verification  needs  (Source:  Task  4.2.1) 

•  M&S  design  products  (Source:  Task  4.2.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  in  the  input  design 

4.2.7  Analyze  the  M&S  design  products  to  verify  the  output  functionality. 

Information  Required: 

•  M&S  design  product  verification  needs  (Source:  Task  4.2.1) 
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•  M&S  design  products  (Source:  Task  4.2.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  in  the  output  design 

4.2.8  Analyze  the  inconsistencies  discovered  during  the  M&S  design  product  verification  to  develop  an 
integrated  description  of  the  M&S  design  product  inconsistencies. 

Information  Required: 

•  Inconsistencies  discovered  within  the  M&S  design  products  (Source:  Task  4.2.4) 

•  Inconsistencies  discovered  between  the  M&S  design  products  (Source:  Task  4.2.5) 

•  Inconsistencies  discovered  in  the  input  design  (Source:  Task  4.2.6) 

•  Inconsistencies  discovered  in  the  output  design  (Source:  Task  4.2.7) 

Information  Produced: 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  design  products 

•  Verified  M&S  design  products 

4.2.9  Update  the  requirements  traceability  matrix  with  the  evidence  from  design  verification. 
Information  Required: 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  design  products 
(Source:  Task  4.2.8) 

•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Requirements  traceability  matrix  updated  with  the  design  verification  evidence 


Activity  4.3  -  Verify  the  M&S  Object  Code 

Activity  Description:  In  this  activity,  the  V&V  Team  verifies  the  M&S  object  code  if  needed. 

Activity  Tailoring  Guidance:  This  activity  need  not  be  performed  if  the  supplemental  verification  needs  do 
not  include  any  needs  for  verifying  the  M&S  object  code  (e.g.,  the  Developer  has  already  performed  sufficient 
verification  of  the  M&S  object  code). 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  All  object  code  products  can  only  be  analyzed  for  inconsistencies  and  cannot  be  tested 
because  they  cannot  be  executed  (i.e.,  only  the  M&S  executable  can  be  executed). 

Task  Descriptions: 

4.3.1  Analyze  the  supplemental  verification  needs  to  determine  the  needs  for  verifying  the  M&S  object 
code. 
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Information  Required: 

•  Supplemental  verification  needs  (Source:  Activity  4.1) 

Information  Produced: 

•  M&S  object  code  verification  needs 

4.3.2  Obtain  the  M&S  object  code  from  the  Developer. 

Information  Required: 

•  M&S  object  code  (Source:  Developer) 

Information  Produced: 

•  M&S  object  code 

4.3.3  Analyze  the  M&S  object  code  for  the  possible  existence  of  computational  anomalies. 
Information  Required: 

•  M&S  object  code  verification  needs  (Source:  Task  4.3.1) 

•  M&S  object  code  (Source:  Task  4.3.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Computational  anomalies  discovered  in  the  M&S  object  code 

4.3.4  Analyze  the  M&S  object  code  to  determine  its  internal  consistency. 

Information  Required: 

•  M&S  object  code  verification  needs  (Source:  Task  4.3.1) 

•  M&S  object  code  (Source:  Task  4.3.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  within  the  M&S  object  code 

4.3.5  Analyze  the  M&S  object  code  to  verify  its  input  functionality. 

Information  Required: 

•  M&S  object  code  verification  needs  (Source:  Task  4.3.1) 

•  M&S  object  code  (Source:  Task  4.3.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  in  the  object  code  input  functionality 

4.3.6  Analyze  the  M&S  object  code  to  verify  its  output  functionality. 

Information  Required: 

•  M&S  object  code  verification  needs  (Source:  Task  4.3.1) 
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•  M&S  object  code  (Source:  Task  4.3.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  in  the  object  code  output  functionality 

4.3.7  Analyze  the  verified  M&S  object  code  and  the  verified  M&S  design  products  to  determine  the 
consistency  between  them  and  to  produce  the  verified  symbolic  code. 

Information  Required: 

•  M&S  object  code  verification  needs  (Source:  Task  4.3.1) 

•  M&S  object  code  (Source:  Task  4.3.2) 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  between  the  verified  M&S  design  products  and  the  M&S  object  code 

4.3.8  Analyze  the  inconsistencies  discovered  during  the  M&S  object  code  verification  to  develop  an  integrated 
description  of  the  M&S  object  code  inconsistencies. 

Information  Required: 

•  Inconsistencies  discovered  within  the  M&S  object  code  (Source:  Task  4.3.4) 

•  Inconsistencies  discovered  in  the  object  code  input  functionality  (Source:  Task  4.3.5) 

•  Inconsistencies  discovered  in  the  object  code  output  functionality  (Source:  Task  4.3.6) 

•  Inconsistencies  discovered  between  the  verified  M&S  design  products  and  the  M&S  object  code 
(Source:  Task  4.3.7) 

Information  Produced: 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  object  code 

•  Verified  M&S  object  code 

4.3.9  Update  the  requirements  traceability  matrix  with  the  evidence  from  object  code  verification. 
Information  Required: 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  object  code 
(Source:  Task  4.3.8) 

•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Requirements  traceability  matrix  updated  with  the  M&S  object  code  verification  evidence 

Activity  4.4  -  Verify  the  M&S  Executable 

Activity  Description:  The  V&V  Team  verifies  the  M&S  executable  in  this  activity.  Depending  upon  the 
available  tools  and  personnel  skill,  verification  of  the  M&S  executable  can  be  done  through  analysis,  testing 
or  both. 
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Activity  Tailoring  Guidance: 

•  This  activity  need  not  be  perfonned  if  the  supplemental  verification  needs  do  not  include  any  needs 
for  verifying  the  M&S  executable  (i.e.,  the  Developer  has  already  performed  sufficient  verification  of 
the  M&S  executable). 

•  This  activity  can  only  be  performed  if  the  M&S  executable  is  available  from  the  Developer. 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions: 

•  The  only  development  product  that  can  be  reasonably  tested  is  the  M&S  executable. 

•  Analysis  of  the  M&S  executable  is  only  possible  if  done  with  a  suitable  tool. 

•  Verification  testing  is  performed  as  an  activity  separate  from  M&S  results  validation  testing. 

Task  Descriptions: 

4.4. 1  Analyze  the  supplemental  verification  needs  to  determine  the  needs  for  verifying  the  M&S  executable. 
Information  Required: 

•  Supplemental  verification  needs  (Source:  Activity  4.1) 

Information  Produced: 

•  M&S  executable  verification  needs 

4.4.2  Obtain  the  M&S  executable  from  the  Developer. 

Information  Required: 

•  M&S  executable  (Source:  Developer) 

Information  Produced: 

•  M&S  executable 

4.4.3  Analyze  the  M&S  executable  for  computational  anomalies. 

Information  Required: 

•  M&S  executable  verification  needs  (Source:  Task  4.4.1) 

•  M&S  executable  (Source:  Task  4.4.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Computational  anomalies  in  the  M&S  executable  discovered  through  analysis 

4.4.4  Analyze  the  M&S  executable  to  determine  its  internal  consistency. 

Information  Required: 

•  M&S  executable  verification  needs  (Source:  Task  4.4.1) 
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•  M&S  executable  (Source:  Task  4.4.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  within  the  M&S  executable  discovered  through  analysis 

4.4.5  Analyze  the  M&S  executable  and  the  verified  M&S  object  code  to  determine  the  consistency  between 
them. 

Information  Required: 

•  M&S  executable  verification  needs  (Source:  Task  4.4.1) 

•  M&S  executable  (Source:  Task  4.4.2) 

•  Verification  M&S  object  code  (Source:  Activity  4.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  between  the  M&S  executable  and  the  verified  M&S  object  code  discovered 
through  analysis 

4.4.6  Analyze  the  M&S  executable  verification  needs,  the  inconsistencies  found  in  the  M&S  design  products 
and  the  M&S  object  code,  the  inconsistencies  found  in  the  M&S  executable  through  analysis  and  the 
V&V  Plan  to  develop  the  M&S  executable  verification  testing  plan. 

Information  Required: 

•  M&S  executable  verification  needs  (Source:  Task  4.4.1) 

•  Inconsistencies  found  in  the  M&S  design  products  (Source:  Activity  4.2) 

•  Inconsistencies  found  in  the  M&S  object  code  (Source:  Activity  4.3) 

•  Computational  anomalies  in  the  M&S  executable  discovered  through  analysis  (Source:  Task  4.4.3) 

•  Inconsistencies  within  the  M&S  executable  discovered  through  analysis  (Source:  Task  4.4.4) 

•  Inconsistencies  between  the  M&S  executable  and  the  verified  M&S  object  code  discovered 
through  analysis  (Source:  Task  4.4.5) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  executable  verification  testing  plan 

4.4.7  Analyze  the  M&S  executable  verification  testing  plan  and  the  verified  M&S  design  products  to  identify 
verification  test  cases  and  procedures. 

Information  Required: 

•  M&S  executable  verification  testing  plan  (Source:  Task  4.4.6) 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  executable  verification  test  cases  and  procedures 

4.4.8  Test  the  M&S  executable  with  the  verification  test  cases  and  collect  the  results  specified  in  the  M&S 
executable  verification  testing  plan. 
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Information  Required: 

•  M&S  executable  verification  testing  plan  (Source:  Task  4.4.6) 

•  M&S  executable  verification  test  cases  and  procedures  (Source:  Task  4.4.7) 

•  M&S  executable  (Source:  Task  4.4.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  executable  verification  testing  results 

4.4.9  Characterize  the  conditions  under  which  the  M&S  executable  verification  testing  was  performed. 
Information  Required: 

•  M&S  executable  verification  testing  plan  (Source:  Task  4.4.6) 

•  M&S  executable  verification  test  cases  and  procedures  (Source:  Task  4.4.7) 

•  M&S  executable  (Source:  Task  4.4.2) 

•  M&S  executable  verification  testing  results  (Source:  Task  4.4.8) 

Information  Produced: 

•  M&S  executable  verification  testing  conditions 

4.4. 10  Analyze  the  M&S  executable  verification  testing  results  for  computational  anomalies. 

Information  Required: 

•  M&S  executable  verification  testing  results  (Source:  Task  4.4.8) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Computational  anomalies  in  the  M&S  executable  discovered  through  testing 

4.4. 1 1  Analyze  the  M&S  executable  verification  testing  results  to  determine  the  inconsistencies  found  in  the 
M&S  executable  through  testing. 

Information  Required: 

•  M&S  executable  verification  testing  results  (Source:  Task  4.4.8) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  in  the  M&S  executable  discovered  through  testing 

4.4. 12  Analyze  the  M&S  executable  verification  testing  results  to  identify  interoperability  and  compatibility 
problems. 

Information  Required: 

•  M&S  executable  verification  testing  results  (Source:  Task  4.4.8) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Interoperability  and  compatibility  problems  discovered  through  testing 


RTO-TR-MSG-054 


A1  -47 


APPENDIX  1  -  VERIFICATION  AND  VALIDATION  (V&V)  COMPOSITE  MODEL 


ORGANIZATION 


4.4.13  Analyze  the  computational  anomalies  discovered  during  the  M&S  executable  verification  to  develop 
an  integrated  description  of  the  M&S  executable  computational  anomalies. 

Information  Required: 

•  Computational  anomalies  in  the  M&S  executable  discovered  through  analysis 
(Source:  Task  4.4.3) 

•  Computational  anomalies  in  the  M&S  executable  discovered  through  testing  (Source:  Task  4.4. 10) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  computational  anomalies  in  the  M&S  executable 

4.4. 14  Analyze  the  inconsistencies  and  interoperability  and  compatibility  problems  discovered  during 
the  M&S  executable  verification  to  develop  an  integrated  description  of  the  M&S  executable 
inconsistencies. 

Information  Required: 

•  Inconsistencies  within  the  M&S  executable  discovered  through  analysis  (Source:  Task  4.4.4) 

•  Inconsistencies  between  the  M&S  executable  and  the  verified  M&S  object  code  discovered 
through  analysis  (Source:  Task  4.4.5) 

•  Inconsistencies  in  the  M&S  executable  discovered  through  testing  (Source:  Task  4.4.1 1) 

•  Interoperability  and  compatibility  problems  discovered  through  testing  (Source:  Task  4.4.12) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  executable 

•  Verified  M&S  executable 

4.4. 15  Update  the  requirements  traceability  matrix  with  the  evidence  from  M&S  executable  verification. 
Information  Required: 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  executable 
(Source:  Task  4.4. 14) 

•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Requirements  traceability  matrix  updated  with  the  M&S  executable  verification  evidence 

Activity  4.5  -  Verify  the  Development  Products  for  Standards  Compliance 

Activity  Description:  The  V&V  Team  verifies  the  compliance  of  all  of  the  development  products  against  the 
standards  that  have  been  chosen  to  guide  the  development  process. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 
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Activity  Assumptions:  The  User  specifies  any  standards  that  have  been  chosen  to  guide  the  development  process. 

Task  Descriptions: 

4.5.1  Obtain  the  results  of  the  Developer’s  standards  compliance  verification  efforts. 

Information  Required: 

•  Standards  compliance  verification  results  from  the  Developer  (Source:  Developer) 

Information  Produced 

•  Standards  compliance  verification  results 

4.5.2  Analyze  the  supplemental  verification  needs  and  the  standards  compliance  verification  results  from 
the  Developer  to  determine  the  needs  for  standards  compliance  verification. 

Information  Required: 

•  Supplemental  verification  needs  (Source:  Activity  4.1) 

•  Standards  compliance  verification  results  from  the  Developer  (Source:  Activity  4.1) 

Information  Produced: 

•  Standards  compliance  verification  needs 

4.5.3  Analyze  the  standards  compliance  verification  needs  and  the  M&S  requirements  to  determine  the 
applicable  standards  needs. 

Information  Required: 

•  Standards  compliance  verification  needs  (Source:  Task  4.5.2) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Applicable  standards  needs 

4.5.4  Obtain  the  standards  defined  by  the  applicable  standards  needs  from  the  standards  organizations. 
Information  Required: 

•  Applicable  standards  needs  (Source:  Task  4.5.3) 

•  Library  of  standards  (Source:  Standards  organizations) 

Information  Produced: 

•  Applicable  standards 

4.5.5  Analyze  the  M&S  design  products  against  the  applicable  standards  to  determine  their  compliance. 
Information  Required: 

•  Standards  compliance  verification  needs  (Source:  Task  4.5.2) 

•  Validated  and  verified  M&S  conceptual  model  (Source:  Activity  3.2) 

•  Verified  and  validated  collected  scenarios  (Source:  Activity  3.3) 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

•  Verified  M&S  Object  Code  (Source:  Activity  4.3) 
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•  Verified  M&S  Executable  (Source:  Activity  4.4) 

•  Applicable  standards  (Source:  Task  4.5.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Compliance  of  the  M&S  design  products  with  the  applicable  standards 

4.5.6  Analyze  the  M&S  object  code  against  the  applicable  standards  to  determine  its  compliance. 
Information  Required: 

•  Standards  compliance  verification  needs  (Source:  Task  4.5.2) 

•  Verified  M&S  object  code  (Source:  Activity  4.3) 

•  Applicable  standards  (Source:  Task  4.5.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Compliance  of  the  M&S  object  code  with  the  applicable  standards 

4.5.7  Test  the  M&S  executable  against  the  applicable  standards  to  determine  its  compliance. 

Information  Required: 

•  Standards  compliance  verification  needs  (Source:  Task  4.5.2) 

•  Verified  M&S  executable  (Source:  Activity  4.4) 

•  Applicable  standards  (Source:  Task  4.5.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Compliance  of  the  M&S  executable  with  the  applicable  standards 

4.5.8  Analyze  the  compliance  of  the  M&S  design  products,  M&S  object  code,  and  M&S  executable  with  the 
applicable  standards  to  develop  an  integrated  picture  of  M&S  development  product  compliance  with 
applicable  standards. 

Information  Required: 

•  Compliance  of  the  M&S  design  products  with  the  applicable  standards  (Source:  Task  4.5.5) 

•  Compliance  of  the  M&S  object  code  with  the  applicable  standards  (Source:  Task  4.5.6) 

•  Compliance  of  the  M&S  executable  with  the  applicable  standards  (Source:  Task  4.5.7) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  M&S  development  product  compliance  with  applicable  standards 

4.5.9  Update  the  requirements  traceability  matrix  with  the  evidence  from  verification  against  applicable 
standards. 

Information  Required: 

•  Integrated  description  of  the  M&S  development  product  compliance  with  applicable  standards 
(Source:  Task  4.5.8) 


A1  -  50 


RTO-TR-MSG-054 


APPENDIX  1  -  VERIFICATION  AND  VALIDATION  (V&V)  COMPOSITE  MODEL 


•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Requirements  traceability  matrix  updated  with  the  standards  verification  evidence 


Activity  4.6  -  Integrate  the  Supplemental  Verification  Evidence 

Activity  Description:  This  activity  collects  and  integrates  the  evidence  produced  by  the  supplemental  verification 
activities. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  The  M&S  configuration  management  history  has  been  obtained  in  a  previous  activity. 

Task  Descriptions: 

4.6.1  Analyze  the  Developer’s  verification  products  and  the  M&S  configuration  management  history  to 
produce  a  description  of  the  computational  anomalies  discovered  by  the  Developer’s  verification. 

Information  Required: 

•  Developer’s  verification  products  (Source:  Activity  4.1) 

•  M&S  configuration  management  history  (Source:  Activity  4.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Description  of  the  computational  anomalies  discovered  by  the  Developer’s  verification 

4.6.2  Analyze  the  computational  anomalies  discovered  by  the  supplemental  verification  of  the  design, 
object  and  executable  products  and  the  Developer’s  verification  products  to  produce  an  integrated 
description  of  the  computational  anomalies  discovered  by  the  supplemental  verification. 

Information  Required: 

•  Description  of  the  computational  anomalies  discovered  by  the  Developer’s  verification 
(Source:  Task  4.6.1) 

•  Computational  anomalies  discovered  in  the  M&S  design  products  (Source:  Activity  4.2) 

•  Computational  anomalies  discovered  in  the  M&S  object  code  (Source:  Activity  4.3) 

•  Integrated  description  of  the  computational  anomalies  discovered  in  the  M&S  executable 
(Source:  Activity  4.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  computational  anomalies  discovered  in  the  M&S  development  products 

4.6.3  Analyze  the  Developer’s  verification  products  and  the  M&S  configuration  management  history  to 
produce  a  description  of  the  inconsistencies  discovered  by  the  Developer’s  verification. 
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Information  Required: 

•  Developer’s  verification  products  (Source:  Activity  4.1) 

•  M&S  configuration  management  history  (Source:  Activity  4.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  by  the  Developer’s  verification 

4.6.4  Analyze  the  inconsistencies  found  in  the  M&S  design  products,  symbolic  code  and  executable  to 
produce  an  integrated  description  of  the  inconsistencies  in  the  M&S  development  products. 

Information  Required: 

•  Inconsistencies  discovered  by  the  Developer’s  verification  (Source:  Task  4.6.3) 

•  Inconsistencies  discovered  within  the  M&S  design  products  (Source:  Activity  4.2) 

•  Inconsistencies  discovered  between  the  M&S  design  products  (Source:  Activity  4.2) 

•  Inconsistencies  discovered  within  the  M&S  object  code  (Source:  Activity  4.3) 

•  Inconsistencies  discovered  between  the  verified  M&S  design  products  and  the  M&S  object  code 
(Source:  Activity  4.3) 

•  Inconsistencies  discovered  within  the  M&S  executable  (Source:  Activity  4.4) 

•  Inconsistencies  discovered  between  the  verified  M&S  symbolic  code  and  the  M&S  executable 
(Source:  Activity  4.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  development  products 

4.6.5  Prepare  citations  of  the  verification  documentation  and  resolve  any  citation  problems. 

Information  Required: 

•  M&S  configuration  management  history  (Source:  Activity  4.1) 

•  Developer’s  verification  products  (Source:  Activity  4.1) 

•  M&S  design  products  (Source:  Activity  4.2) 

•  M&S  object  code  (Source:  Activity  4.3) 

•  M&S  executable  (Source:  Activity  4.4) 

•  Standards  compliance  verification  results  from  the  Developer  (Source:  Activity  4.5) 

Information  Produced: 

•  Integrated  citations  supporting  the  development  of  the  verification  evidence 


A1  -  52 


RTO-TR-MSG-054 


APPENDIX  1  -  VERIFICATION  AND  VALIDATION  (V&V)  COMPOSITE  MODEL 


A1.6  PHASE  5  -  APPLY  THE  VERIFICATION  PRODUCTS  TO  VALIDATION 

Activity  5.1  -  Determine  the  Effects  of  Developmental  Product  Inconsistencies 

Activity  Description:  This  activity  verifies  the  consistency  of  the  development  products  against  the  verified 
and  validated  M&S  conceptual  model. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions: 

•  An  M&S  conceptual  model  exists  and  that  it  has  been  validated  against  the  acceptability  criteria. 

•  None  of  the  inconsistencies  discovered  through  the  verification  of  the  development  products  have 
been  corrected  and  therefore  their  effects  still  exist  in  the  M&S  executable. 

Task  Descriptions: 

5.1.1  Analyze  the  supplemental  verification  needs  to  determine  the  needs  for  verifying  against  the  M&S 
conceptual  model. 

Information  Required: 

•  Supplemental  verification  needs  (Source:  Activity  4.1) 

Information  Produced: 

•  Needs  for  verifying  against  the  M&S  conceptual  model 

5.1.2  Analyze  the  computational  anomalies  discovered  in  the  M&S  conceptual  model  and  the  combined 
verification  products  with  the  M&S  conceptual  model  to  determine  the  possible  effects  of  computational 
anomalies  discovered. 

Information  Required: 

•  Needs  for  verifying  against  the  M&S  conceptual  model  (Source:  Task  5.1.1) 

•  Computational  anomalies  discovered  in  the  M&S  conceptual  model  (Source:  Activity  3.1) 

•  Integrated  description  of  the  computational  anomalies  discovered  in  the  M&S  development 
products  (Source:  Activity  4.6) 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Possible  effects  of  computational  anomalies  upon  the  M&S  capabilities  and  limitations 

5.1.3  Analyze  the  inconsistencies  discovered  in  the  M&S  development  products  with  the  M&S  conceptual 
model  to  determine  the  possible  effects  of  those  inconsistencies. 

Information  Required: 

•  Needs  for  verifying  against  the  M&S  conceptual  model  (Source:  Task  5.1.1) 
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•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  conceptual  model 
(Source:  Activity  3.1) 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  combined  verification  effort 
(Source:  Activity  4.6) 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Possible  effects  of  inconsistencies  within  the  M&S  development  products  upon  the  M&S 
capabilities  and  limitations 

5.1.4  Analyze  the  verified  M&S  design  products  and  the  M&S  conceptual  model  to  identify  any 
inconsistencies  between  them. 

Information  Required: 

•  Needs  for  verifying  against  the  M&S  conceptual  model  (Source:  Task  5.1.1) 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  discovered  between  the  M&S  conceptual  model  and  the  design  products 

•  Verified  and  validated  M&S  design  products 

5.1.5  Analyze  the  inconsistencies  discovered  between  the  M&S  conceptual  model  and  the  M&S  development 
products  with  the  M&S  conceptual  model  to  determine  the  possible  effects  of  those  inconsistencies. 

Information  Required: 

•  Needs  for  verifying  against  the  M&S  conceptual  model  (Source:  Task  5.1.1) 

•  Inconsistencies  discovered  between  the  M&S  conceptual  model  and  the  design  products 
(Source:  Task  5.1.4) 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  development  products 
(Source:  Activity  4.6) 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Possible  effects  of  inconsistencies  between  the  M&S  conceptual  model  and  the  development 
products  upon  the  M&S  capabilities  and  limitations 

5.1.6  Analyze  the  possible  effects  of  computational  anomalies  and  development  product  inconsistencies 
upon  the  M&S  capabilities  and  limitations  to  determine  the  capabilities  and  limitations  of  the  verified 
M&S  executable. 

Information  Required: 

•  Possible  effects  of  computational  anomalies  upon  the  M&S  capabilities  and  limitations 
(Source:  Task  5.1.2) 

•  Possible  effects  of  inconsistencies  within  the  M&S  development  products  upon  the  M&S 
capabilities  and  limitations  (Source:  Task  5.1.3) 


A1  -  54 


RTO-TR-MSG-054 


APPENDIX  1  -  VERIFICATION  AND  VALIDATION  (V&V)  COMPOSITE  MODEL 


•  Possible  effects  of  inconsistencies  between  the  M&S  conceptual  model  and  the  development 
products  upon  the  M&S  capabilities  and  limitations  (Source:  Task  5.1.5) 

Information  Produced: 

•  Description  of  the  capabilities  and  limitations  of  the  verified  M&S  executable 


Activity  5.2  -  Evaluate  M&S  Validity  from  the  Verification  Products 

Activity  Description:  This  activity  compares  the  description  of  M&S  capabilities  and  limitations  of  the  verified 
M&S  executable  with  the  acceptability  criteria  to  identify  the  needed  capabilities  that  the  M&S  executable  could 
support  and  the  limitations  implied  by  the  development  product  verification  evidence,  and  estimate  the  coverage 
and  uncertainties  in  the  development  product  verification  evidence. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  (Blank) 

Descriptions: 

Analyze  the  description  of  the  capabilities  and  limitations  of  the  verified  M&S  executable  and  the 
acceptability  criteria  that  define  the  required  functionality  to  determine  the  functionality  needs  that  are 
met,  identify  those  criteria  that  are  not  met,  and  identify  those  criteria  for  which  insufficient  information 
is  available. 

Information  Required: 

•  Description  of  the  capabilities  and  limitations  of  the  verified  M&S  executable  (Source:  Activity  5.1) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Acceptability  criteria  that  define  the  required  functionality  that  the  verified  M&S  executable 
meets 

•  Acceptability  criteria  that  define  the  required  functionality  that  the  verified  M&S  executable  does 
not  meet 

•  Acceptability  criteria  that  define  the  required  functionality  for  which  insufficient  information  is 
available  from  the  verified  M&S  executable 

5.2.2  Analyze  the  description  of  the  capabilities  and  limitations  of  the  verified  M&S  executable  and  the 

acceptability  criteria  that  define  the  required  quality  to  determine  the  quality  needs  that  are  met,  identify 
those  criteria  that  are  not  met,  and  identify  those  criteria  for  which  insufficient  information  is  available. 

Information  Required: 

•  Description  of  the  capabilities  and  limitations  of  the  verified  M&S  executable  (Source:  Activity  5.1) 

•  Acceptability  criteria  defining  needed  qualify  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 


Task 

5.2.1 
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Information  Produced: 

•  Acceptability  criteria  that  define  the  required  quality  that  the  verified  M&S  executable  meets 

•  Acceptability  criteria  that  define  the  required  quality  that  the  verified  M&S  executable  does  not  meet 

•  Acceptability  criteria  that  define  the  required  quality  for  which  insufficient  information  is 
available  from  the  verified  M&S  executable 

5.2.3  Analyze  the  acceptability  criteria  that  the  M&S  executable  meets  and  the  M&S  conceptual  model  to 
determine  the  needed  M&S  capabilities  that  the  M&S  executable  could  support. 

Information  Required: 

•  Acceptability  criteria  that  define  the  required  functionality  that  the  verified  M&S  executable 
meets  (Source:  Task  5.3.1) 

•  Acceptability  criteria  that  define  the  required  quality  that  the  verified  M&S  executable  meets 
(Source:  Task  5.3.2) 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.1) 

Information  Produced: 

•  Needed  M&S  capabilities  that  the  M&S  executable  could  support 

5.2.4  Analyze  the  acceptability  criteria  that  the  M&S  executable  does  not  meet  and  the  M&S  conceptual 
model  to  determine  the  M&S  limitations  implied  by  the  M&S  executable  verification. 

Information  Required: 

•  Acceptability  criteria  that  define  the  required  functionality  that  the  verified  M&S  executable  does 
not  meet  (Source:  Task  5.2.1) 

•  Acceptability  criteria  that  define  the  required  quality  that  the  verified  M&S  executable  does  not 
meet  (Source:  Task  5.2.2) 

•  Verified  and  validated  M&S  conceptual  model  (Source:  Activity  3.1) 

Information  Produced: 

•  M&S  limitations  implied  by  the  development  product  verification  evidence 

•  Verified  and  validated  M&S  executable 

5.2.5  Analyze  the  acceptability  criteria  for  which  insufficient  information  is  available  to  determine  the 
information  gaps  in  the  evidence  from  the  development  product  verification. 

Information  Required: 

•  Acceptability  criteria  that  define  the  required  functionality  for  which  insufficient  information  is 
available  from  the  verified  M&S  executable  (Source:  Task  5.2.1) 

•  Acceptability  criteria  that  define  the  required  quality  for  which  insufficient  information  is 
available  from  the  verified  M&S  executable  (Source:  Task  5.2.2) 

Information  Produced: 

•  Information  gaps  in  the  development  product  verification  evidence 

5.2.6  Analyze  the  M&S  capabilities,  limitations  and  information  gaps  implied  by  the  development  product 
verification  evidence  and  the  acceptability  criteria  to  estimate  the  coverage  of  the  development  product 
verification  evidence. 
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Information  Required: 

•  Needed  M&S  capabilities  that  the  M&S  executable  could  support  (Source:  Task  5.2.3) 

•  M&S  limitations  implied  by  the  development  product  verification  evidence  (Source:  Task  5.2.4) 

•  Information  gaps  in  the  development  product  verification  evidence  (Source:  Task  5.2.5) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  coverage  of  the  development  product  verification  evidence 

5.2.7  Analyze  the  M&S  capabilities  and  limitations  implied  by  the  development  product  verification  evidence, 
the  information  gaps  in  the  development  product  verification  evidence  and  the  estimated  coverage  of 
the  development  product  verification  evidence  to  estimate  the  uncertainties  in  the  development  product 
verification  evidence. 

Information  Required: 

•  Needed  M&S  capabilities  that  the  M&S  executable  could  support  (Source:  Task  5.2.3) 

•  M&S  limitations  implied  by  the  development  product  verification  evidence  (Source:  Task  5.2.4) 

•  Information  gaps  in  the  development  product  verification  evidence  (Source:  Task  5.2.5) 

•  Estimated  coverage  of  the  development  product  verification  evidence  (Source:  Task  5.2.6) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  uncertainties  in  the  development  product  verification  evidence 

5.2.8  Update  the  requirements  traceability  matrix  with  the  development  product  verification  evidence. 
Information  Required: 

•  Needed  M&S  capabilities  that  the  M&S  executable  could  support  (Source:  Task  5.2.3) 

•  M&S  limitations  implied  by  the  development  product  verification  evidence  (Source:  Task  5.2.4) 

•  Information  gaps  in  the  development  product  verification  evidence  (Source:  Task  5.2.5) 

•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Requirements  traceability  matrix  updated  with  the  development  product  verification  evidence 


Activity  5.3  -  Employ  the  Verification  Evidence 

Activity  Description:  This  activity  employs  the  development  product  verification  evidence  to  identify  factors 
that  may  constrain  use  and  candidate  output  sampling  areas. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 
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Activity  Assumptions:  (Blank) 

Task  Descriptions: 

5.3.1  Analyze  the  M&S  limitations  implied  by  the  development  product  verification  evidence,  the  information 
gaps  in  the  development  product  verification  evidence  and  the  acceptability  criteria  to  identify  the  factors 
that  may  constrain  use. 

Information  Required: 

•  M&S  limitations  implied  by  the  development  product  verification  evidence  (Source:  Activity  5.2) 

•  Information  gaps  in  the  development  product  verification  evidence  (Source:  Activity  5.2) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Factors  that  may  constrain  use  implied  by  the  development  product  verification  evidence 

5.3.2  Analyze  the  M&S  capabilities  and  limitations  implied  by  the  development  product  verification  evidence, 
the  information  gaps  in  the  development  product  verification  evidence  and  the  acceptability  criteria  to 
identify  the  candidate  output  sampling  areas  suggested  by  the  development  product  verification  evidence. 

Information  Required: 

•  Needed  M&S  capabilities  that  the  M&S  executable  could  support  (Source:  Activity  5.2) 

•  M&S  limitations  implied  by  the  development  product  verification  evidence  (Source:  Activity  5.2) 

•  Information  gaps  in  the  development  product  verification  evidence  (Source:  Activity  5.2) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Candidate  output  sampling  areas  implied  by  the  development  product  verification  evidence 
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A1.7  PHASE  6  -  VERIFY  AND  VALIDATE  THE  DATA  AND  KNOWLEDGE  SETS 

Activity  6.1  -  Identify  the  Data  and  Knowledge  Sources  and  Their  Pedigrees 

Activity  Description:  All  M&S  consist  of  the  executable  or  computational  models  and  the  data  or  knowledge 
that  those  models  require.  In  this  activity,  the  V&V  practitioner  identifies  the  different  data  and  knowledge 
sets  that  an  M&S  needs,  the  available  sources  of  those  data  sets  and  the  pedigrees  of  those  sources. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  This  activity  has  tasks  for  collecting  the  descriptions  of  the  data  and  knowledge  and  their 
sources  (i.e.,  meta-data)  because  these  tasks  take  time  and  the  required  information  is  commonly  not  easily 
available  in  a  single  place. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

6.1.1  Obtain  the  M&S  data  model. 

Information  Required: 

•  M&S  data  model  (Source:  Developer) 

Information  Produced: 

•  M&S  data  model 

6. 1 .2  Analyze  the  M&S  data  model  to  determine  what  data  and  knowledge  sets  the  M&S  needs  to  support 
the  intended  use. 

Information  Required: 

•  M&S  data  model  (Source:  Task  6.1.1) 

•  Statement  of  intended  use  (Source:  Preceding  accreditation  activity) 

Information  Produced: 

•  Data  and  knowledge  sets  needed  to  support  the  intended  use 

6. 1 .3  Obtain  the  available  data  and  knowledge  sets  needed  to  support  the  intended  use. 

Information  Required: 

•  Available  data  and  knowledge  sets  needed  to  support  the  intended  use  (Source:  External  data 
sources) 

Information  Produced: 

•  Available  data  and  knowledge  sets 

6. 1 .4  Obtain  descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use. 

Information  Required: 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use 
(Source:  External  data  sources) 
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Information  Produced: 

•  Descriptions  of  the  data  and  knowledge  sets 

6. 1 .5  Obtain  the  V&V  history  for  the  available  data  and  knowledge  sets. 

Information  Required: 

•  V&V  history  for  the  available  data  and  knowledge  sets  (Source:  External  data  sources) 
Information  Produced: 

•  V&V  history  for  the  available  data  and  knowledge  sets 

6. 1 .6  Analyze  the  descriptions  of  the  available  data  and  knowledge  sets  to  identify  their  sources  if  possible. 
Information  Required: 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use 
(Source:  Task  6.1.4) 

Information  Produced: 

•  Possible  sources  of  the  available  data  and  knowledge  sets 

6. 1 .7  Obtain  descriptions  of  the  identified  sources  for  the  available  data  and  knowledge  sets. 

Information  Required: 

•  Descriptions  of  the  data  and  knowledge  set  sources  (Source:  External  data  sources) 
Information  Produced: 

•  Descriptions  of  the  data  and  knowledge  set  sources 

6.1.8  Analyze  the  descriptions  of  the  data  and  knowledge  set  sources  to  identify  the  pedigrees  of  those 
sources. 

Information  Required: 

•  Descriptions  of  the  data  and  knowledge  set  sources  (Source:  Task  6.1.7) 

Information  Produced: 

•  Data  and  knowledge  set  source  pedigrees 

6.1.9  Analyze  the  descriptions  of  the  available  data  and  knowledge  sets  and  their  sources  to  identify  the 
data  and  knowledge  sets  for  which  the  sources  cannot  be  identified. 

Information  Required: 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use 
(Source:  Task  6.1.4) 

•  Descriptions  of  the  data  and  knowledge  set  sources  (Source:  Task  6. 1 .7) 

Information  Produced: 

•  Data  and  knowledge  set  sources  that  cannot  be  identified 
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6. 1. 10  Analyze  the  descriptions  of  the  available  data  and  knowledge  sets  and  their  sources  to  identify  the 
data  and  knowledge  sets  that  come  from  sources  with  no  pedigrees. 

Information  Required: 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use 
(Source:  Task  6.1.4) 

•  Descriptions  of  the  data  and  knowledge  set  sources  (Source:  Task  6. 1 .7) 

Information  Produced: 

•  Data  and  knowledge  set  sources  with  no  pedigrees 

6. 1. 1 1  Analyze  the  M&S  data  model  and  the  available  data  and  knowledge  set  descriptions  to  identify  the 
data  or  knowledge  sets  needed  to  support  the  intended  use  that  are  not  available. 

Information  Required: 

•  M&S  data  model  (Source:  Task  6.1.1) 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use 
(Source:  Task  6.1.4) 

Information  Produced: 

•  Unavailable  data  and  knowledge  sets  needed  to  support  the  intended  use 

6. 1 . 12  Prepare  the  source  reference  list  of  data  and  knowledge  set  pedigrees  and  resolve  any  citation  problems. 

Information  Required: 

•  M&S  data  model  (Source:  Task  6.1.1) 

•  Available  data  and  knowledge  sets  needed  to  support  the  intended  use  (Source:  Task  6. 1 .3) 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use 
(Source:  Task  6.1.4) 

•  V&V  history  for  the  available  data  and  knowledge  set  (Source:  Task  6.1.5) 

•  Descriptions  of  the  data  and  knowledge  set  sources  (Source:  Task  6. 1 .7) 

Information  Produced: 

•  Data  and  knowledge  set  pedigree  citation  list 


Activity  6.2  -  Verify  the  Data  and  Knowledge  Sets 

Activity  Description:  In  this  activity,  the  V&V  practitioner  verifies  the  internal  consistency  of  the  data  and 
knowledge  sets  with  which  the  M&S  will  be  accepted  for  the  intended  use.  Data  and  knowledge  sets  can  be 
very  large  so  the  V&V  practitioner  may  need  to  take  advantage  of  the  numerous  tools  available  for  checking 
data  and  knowledge  set  consistency.  These  tools  may  be  applicable  to  many  different  types  of  data  sets  from 
disparate  sources  or  they  may  be  specific  for  particular  data  sets  from  individual  sources.  The  knowledge- 
based  systems  community  has  produced  several  tools  to  check  the  consistency  of  knowledge  bases.  These  are 
usually  specific  for  the  knowledge  representation  used. 

Activity  Tailoring  Guidance:  (Blank) 
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Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

6.2.1  Analyze  the  data  and  knowledge  sets  needed  to  support  the  intended  use  for  consistency  within  sets 
and  identify  any  inconsistencies  that  exist. 

Information  Required: 

•  Available  data  and  knowledge  sets  needed  to  support  the  intended  use  (Source:  Activity  6.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  within  the  available  data  and  knowledge  sets 

6.2.2  Analyze  the  data  and  knowledge  sets  needed  to  support  the  intended  use  for  consistency  between  sets 
and  identify  any  inconsistencies  that  exist. 

Information  Required: 

•  Available  data  and  knowledge  sets  needed  to  support  the  intended  use  (Source:  Task  6.2.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  between  the  available  data  and  knowledge  sets 

6.2.3  Analyze  the  data  and  knowledge  sets  needed  to  support  the  intended  use  for  consistency  with  the 
M&S  data  model  and  identify  any  inconsistencies  that  exist. 

Information  Required: 

•  Available  data  and  knowledge  sets  needed  to  support  the  intended  use  (Source:  Task  6.2.1) 

•  M&S  data  model  (Source:  Activity  6. 1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  between  the  available  data  and  knowledge  sets  and  the  M&S  data  model 

6.2.4  Analyze  the  M&S  design  products  to  identify  the  data  transformations  used. 

Information  Required: 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

Information  Produced: 

•  Data  transformations  used  in  the  M&S 

6.2.5  Analyze  the  data  and  knowledge  sets  needed  to  support  the  intended  use  and  the  data  transfonnations 
used  to  verify  the  correctness  of  those  transformations  with  that  data. 
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Information  Required: 

•  Available  data  and  knowledge  sets  needed  to  support  the  intended  use  (Source:  Task  6.2.1) 

•  Data  transformations  used  in  the  M&S  (Source:  Task  6.2.5) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Inconsistencies  between  the  data  and  knowledge  sets  and  the  data  transformations  used 

6.2.6  Analyze  the  inconsistencies  discovered  during  the  data  and  knowledge  set  verification  to  develop  an 
integrated  description  of  the  data  and  knowledge  set  inconsistencies. 

Information  Required: 

•  Inconsistencies  within  the  available  data  and  knowledge  sets  (Source:  Task  6.2.1) 

•  Inconsistencies  between  the  available  data  and  knowledge  sets  (Source:  Task  6.2.2) 

•  Inconsistencies  between  the  available  data  and  knowledge  sets  and  the  M&S  data  model 
(Source:  Task  6.2.3) 

•  Inconsistencies  between  the  data  and  knowledge  sets  and  the  data  transformations  used 
(Source:  Task  6.2.5) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  data  and  knowledge  set  inconsistencies 


Activity  6.3  -  Validate  Data  and  Knowledge  Sets  Where  Needed 

Activity  Description:  In  this  activity,  the  V&V  practitioner  validates  the  data  and  knowledge  sets  with  which 
the  M&S  will  be  accepted  for  the  intended  use.  The  V&V  practitioner  should  take  advantage  of  the  pedigree 
information  associated  with  the  data  and  knowledge  sets  to  reduce  the  validation  burden  and  validate  those 
sets  only  where  sufficient  pedigree  information  is  not  available. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes: 

•  Gaps  and  inconsistencies  in  both  the  data  itself  and  the  validation  coverage  of  the  data.  These  are 
different  but  related  concepts. 

•  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified  in  the  V&V 
Plan  to  perform  the  analyses. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

6.3. 1  Analyze  the  sensitivity  of  the  M&S  to  the  data  and  knowledge  sets  to  determine  the  importance  of  the 
data  and  knowledge  coverage. 

Information  Required: 

•  Available  data  and  knowledge  sets  needed  to  support  the  intended  use  (Source:  Activity  6.1) 

•  V&V  Plan  (Source:  Activity  1.6) 
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Information  Produced: 

•  Relative  importance  of  data  and  knowledge  sets  for  the  intended  use 

6.3.2  Analyze  the  descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use, 

their  pedigrees,  their  relative  importance  and  the  integrated  description  of  the  data,  and  knowledge  set 
inconsistencies  to  identify  only  the  data  and  knowledge  sets  that  need  to  be  validated  for  the  current 
M&S  configuration. 

Information  Required: 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use 
(Source:  Activity  6.1) 

•  Data  and  knowledge  set  source  pedigrees  (Source:  Activity  6.1) 

•  Relative  importance  of  data  and  knowledge  sets  for  the  intended  use  (Source:  Task  6.3.1) 

•  Integrated  description  of  the  data  and  knowledge  set  inconsistencies  (Source:  Activity  6.2) 

Information  Produced: 

•  Data  and  knowledge  sets  that  need  to  be  validated  for  the  current  M&S  configuration 

6.3.3  Analyze  the  acceptability  criteria  and  M&S  design  products  to  determine  the  acceptability  criteria  that 
define  the  required  functionality  and  quality  of  the  data  and  knowledge  sets  and  the  acceptability  criteria 
upon  which  the  data  and  knowledge  sets  have  no  impact. 

Information  Required: 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Acceptability  criteria  that  define  the  required  functionality  and  quality  of  the  data  and  knowledge 
sets 

6.3.4  Analyze  the  data  and  knowledge  sets  that  need  to  be  validated  against  the  acceptability  criteria  that 
define  the  functionality  needs  to  determine  the  functionality  needs  that  are  met,  identify  those  criteria 
that  are  not  met,  and  identify  those  criteria  for  which  insufficient  information  is  available. 

Information  Required: 

•  Data  and  knowledge  sets  that  need  to  be  validated  for  the  current  M&S  configuration 
(Source:  Task  6.3.2) 

•  Acceptability  criteria  that  define  the  required  functionality  of  the  data  and  knowledge  sets 
(Source:  Task  6.3.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Acceptability  criteria  that  define  the  required  functionality  that  the  data  and  knowledge  sets  meet 

6.3.5  Analyze  the  data  and  knowledge  sets  that  need  to  be  validated,  the  acceptability  criteria  that  define  the 
quality  needs  and  the  M&S  referent  to  determine  the  quality  needs  that  are  met,  identify  those  criteria 
that  are  not  met  and  identify  those  criteria  for  which  insufficient  information  is  available. 
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Information  Required: 

•  Data  and  knowledge  sets  that  need  to  be  validated  for  the  current  M&S  configuration  and 
intended  use  (Source:  Task  6.3.2) 

•  Acceptability  criteria  that  define  the  required  quality  of  the  data  and  knowledge  sets 
(Source:  Task  6.3.3) 

•  Validation  Referent  (Source:  Preceding  accreditation  activity) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Acceptability  criteria  that  define  the  required  quality  that  the  data  and  knowledge  sets  meet 

6.3.6  Analyze  the  acceptability  criteria  that  the  data  and  knowledge  sets  meet  and  the  M&S  design  products 
to  determine  the  M&S  capabilities  that  valid  data  and  knowledge  sets  support. 

Information  Required: 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

•  Acceptability  criteria  that  define  the  required  functionality  that  the  data  and  knowledge  sets  meet 
(Source:  Task  6.3.4) 

•  Acceptability  criteria  that  define  the  required  quality  that  the  data  and  knowledge  sets  meet 
(Source:  Task  6.3.5) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  capabilities  that  the  valid  data  and  knowledge  sets  can  support 

6.3.7  Analyze  the  acceptability  criteria  that  the  data  and  knowledge  sets  do  not  meet  and  the  M&S  design 
products  to  determine  the  M&S  limitations  implied  by  the  data  and  knowledge  sets. 

Information  Required: 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

•  Acceptability  criteria  that  define  the  required  functionality  that  the  data  and  knowledge  sets  do 
not  meet  (Source:  Task  6.3.4) 

•  Acceptability  criteria  that  define  the  required  quality  that  the  data  and  knowledge  sets  do  not  meet 
(Source:  Task  6.3.5) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  M&S  limitations  implied  by  the  data  and  knowledge  set  validation 

6.3.8  Analyze  the  acceptability  criteria  for  which  insufficient  information  is  available  from  the  data  and 
knowledge  set  validation  and  the  M&S  design  products  to  determine  the  information  gaps  implied  by 
those  sets. 

Information  Required: 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

•  Acceptability  criteria  upon  which  the  data  and  knowledge  sets  have  no  impact  (Source:  Task  6.3.3) 

•  Acceptability  criteria  that  define  the  required  functionality  for  which  insufficient  information  is 
available  (Source:  Task  6.3.4) 
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•  Acceptability  criteria  that  define  the  required  quality  for  which  insufficient  information  is 
available  (Source:  Task  6.3.5) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Information  gaps  implied  by  the  data  and  knowledge  set  validation 


Activity  6.4  -  Integrate  and  Employ  the  Data  and  Knowledge  V&V  Evidence 

Activity  Description:  In  this  activity,  the  V&V  practitioner  integrates  the  evidence  gained  from  the  prior  data 
and  knowledge  V&V  activities  into  an  integrated  description  of  the  M&S’s  capabilities  and  limitations. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

6.4. 1  Analyze  the  descriptions  of  the  pedigreed  data  and  knowledge  sets,  the  integrated  descriptions  of  the 
data  and  knowledge  set  inconsistencies,  the  M&S  capabilities  that  the  data  and  knowledge  sets 
support  to  produce  an  integrated  description  of  the  M&S  capabilities  that  the  data  and  knowledge  sets 
can  support. 

Information  Required: 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use  (Source: 
Activity  6.1) 

•  Integrated  description  of  the  data  and  knowledge  set  inconsistencies  (Source:  Activity  6.2) 

•  M&S  capabilities  that  the  valid  data  and  knowledge  sets  can  support  (Source:  Activity  6.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  M&S  capabilities  that  the  data  and  knowledge  sets  can  support 

6.4.2  Analyze  the  descriptions  of  the  pedigreed  data  and  knowledge  sets,  the  integrated  description  of  the 
data  and  knowledge  set  inconsistencies,  the  M&S  limitations  implied  by  the  data  and  knowledge  set 
validation  to  produce  an  integrated  description  of  the  M&S  limitations  implied  the  data  and  knowledge 
sets. 

Information  Required: 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use 
(Source:  Activity  6.1) 

•  Integrated  description  of  the  data  and  knowledge  set  inconsistencies  (Source:  Activity  6.2) 

•  M&S  limitations  implied  by  the  data  and  knowledge  set  validation  (Source:  Activity  6.3) 

•  V&V  Plan  (Source:  Activity  1.6) 
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Information  Produced: 

•  Integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  V&V 

6.4.3  Analyze  the  descriptions  of  the  pedigreed  data  and  knowledge  sets,  the  M&S  design  products, 

the  acceptability  criteria,  the  integrated  descriptions  of  the  data,  and  knowledge  set  inconsistencies, 
the  information  gaps  implied  by  the  data  and  knowledge  set  validation  to  produce  an  integrated 
description  of  the  information  gaps  implied  by  the  data  and  knowledge  sets. 

Information  Required: 

•  Descriptions  of  the  data  and  knowledge  sets  available  to  support  the  intended  use 
(Source:  Activity  6.1) 

•  M&S  design  products  (Source:  Activity  4.2) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  Integrated  description  of  the  data  and  knowledge  set  inconsistencies  (Source:  Activity  6.2) 

•  Information  gaps  implied  by  the  data  and  knowledge  set  validation  (Source:  Activity  6.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  information  gaps  implied  by  the  data  and  knowledge  set  V&V 

6.4.4  Analyze  the  acceptability  criteria  integrated  description  of  the  M&S  capabilities  that  the  data  and 
knowledge  can  support,  the  integrated  description  of  the  limitations  implied  by  the  data  and  knowledge 
sets  and  the  information  gaps  implied  by  the  data  and  knowledge  set  validation  to  estimate  the  coverage 
of  the  data  and  knowledge  set  V&V  evidence. 

Information  Required: 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  Integrated  description  of  the  M&S  capabilities  that  the  data  and  knowledge  sets  can  support 
(Source:  Task  6.4.1) 

•  Integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  V&V 
(Source:  Task  6.4.2) 

•  Integrated  description  of  the  data  and  knowledge  set  inconsistencies  (Source:  Activity  6.2) 

•  Integrated  description  of  the  information  gaps  implied  by  the  data  and  knowledge  V&V 
(Source:  Task  6.4.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  coverage  of  the  data  and  knowledge  set  V&V  evidence 

6.4.5  Analyze  the  integrated  description  of  the  M&S  capabilities  that  the  data  and  knowledge  sets  can 
support,  the  integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  sets  and 
the  estimated  coverage  of  the  data  and  knowledge  set  V&V  evidence  to  estimate  the  uncertainties  in  the 
data  and  knowledge  set  V&V  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  that  the  data  and  knowledge  sets  can  support 
(Source:  Task  6.4.1) 
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•  Integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  set  V&V 
(Source:  Task  6.4.2) 

•  Estimated  coverage  of  the  data  and  knowledge  set  V&V  evidence  (Source:  Task  6.4.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  uncertainties  in  the  data  and  knowledge  set  V&V  evidence 

6.4.6  Analyze  the  integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  set 
V&V,  the  integrated  description  of  the  information  gaps  implied  by  the  data  and  knowledge  set  V&V, 
and  the  acceptability  criteria  to  identify  the  factors  that  may  constrain  use. 

Information  Required: 

•  Integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  V&V 
(Source:  Task  6.4.2) 

•  Integrated  description  of  the  information  gaps  implied  by  the  data  and  knowledge  set  V&V 
(Source:  Task  6.4.3) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Factors  that  may  constrain  use  implied  by  the  data  and  knowledge  set  V&V 

6.4.7  Analyze  the  integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  set 
V&V,  the  integrated  description  of  the  information  gaps  implied  by  the  data  and  knowledge  set  V&V 
and  the  acceptability  criteria  to  identify  the  candidate  output  sampling  areas  implied  by  the  data  and 
knowledge  set  V&V. 

Information  Required: 

•  Integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  V&V 
(Source:  Task  6.4.2) 

•  Integrated  description  of  the  information  gaps  implied  by  the  data  and  knowledge  set  V&V 
(Source:  Task  6.4.3) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Candidate  output  sampling  areas  implied  by  the  data  and  knowledge  set  V&V 

6.4.8  Update  the  requirements  traceability  matrix  with  the  data  and  knowledge  set  V&V  evidence. 
Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  that  the  data  and  knowledge  sets  can  support 
(Source:  Task  6.4.1) 

•  Integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  V&V 
(Source:  Task  6.4.2) 
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•  Integrated  description  of  the  information  gaps  implied  by  the  data  and  knowledge  set  V&V 
(Source:  Task  6.4.3) 

•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Requirements  traceability  matrix  updated  with  the  data  and  knowledge  set  V&V  evidence 
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A1.8  PHASE  7  -  VALIDATE  THE  M&S  RESULTS 

Activity  7.1  -  Leverage  the  Developer  Test  Results  for  Results  Validation 

Activity  Description:  In  this  activity,  the  V&V  practitioner  uses  the  Developer’s  test  results  to  tailor  the 
results  validation  effort. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes: 

•  Again,  even  though  the  Developer’s  test  plan  and  test  results  are  input  to  this  activity,  it  contains  a 
task  for  collecting  that  information  because  it  is  seldom  presented  to  the  V&V  practitioner  without 
any  action.  Often,  the  V&V  practitioner  must  make  several  requests,  sometimes  through  official 
channels,  follow-up  on  those  requests  and  ensure  that  the  information  that  is  delivered  is  complete 
and  current.  Unlike  the  configuration  management  documentation,  test  results  can  be  managed  by 
several  individuals. 

•  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified  in  the 
V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

7.1.1  Obtain  the  test  plan  from  the  Developer. 

Information  Required 

•  Developer’s  test  plan  (Source:  Developer) 

Information  Produced: 

•  Developer’s  test  plan 

7. 1 .2  Obtain  the  testing  results  from  the  Developer. 

Information  Required 

•  Developer’s  testing  results  (Source:  Developer) 

Information  Produced: 

•  Developer’s  testing  results 

7. 1 .3  Obtain  the  M&S  configuration  management  history  from  the  Developer. 

Information  Required 

•  M&S  configuration  management  history  (Source:  Developer) 

Information  Produced: 

•  M&S  configuration  management  history 
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7. 1 .4  Analyze  the  Developer  testing  results  and  test  plan  and  the  M&S  configuration  management  history  to 
identify  those  test  results  that  are  relevant  to  the  current  M&S  configuration. 

Information  Required: 

•  Developer’s  test  plan  (Source:  Task  7.1.1) 

•  Developer’s  testing  results  (Source:  Task  7. 1 .2) 

•  M&S  configuration  management  history  (Source:  Task  7. 1 .3) 

Information  Produced: 

•  Relevant  Developer’s  testing  results 

7.1.5  Analyze  the  relevant  Developer  testing  results  and  the  acceptability  criteria  to  identify  the  Relevant 
and  applicable  Developer  test  results. 

Information  Required: 

•  Relevant  Developer’s  testing  results  (Source:  Task  7.1.4) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

Information  Produced: 

•  Relevant  and  applicable  Developer’s  testing  results 

7. 1 .6  Analyze  the  Relevant  and  applicable  Developer  testing  results  and  the  acceptability  criteria  to  determine 
the  capabilities  and  limitations  of  the  current  M&S  configuration. 

Information  Required: 

•  Applicable  Developer’s  testing  results  (Source:  Task  7. 1 .5) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  testing 

7.1.7  Analyze  the  M&S  capabilities  and  limitations  from  the  Developer’s  testing  and  the  acceptability 
criteria  to  identify  any  gaps  and  inconsistencies  that  may  exist  in  those  results. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  testing 
(Source:  Task  7.1.6) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Description  of  the  gaps  and  inconsistencies  in  the  Developer’s  testing  results 
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ORGANIZATION 


7.1.8  Analyze  the  description  of  M&S  capabilities  and  limitations  from  the  Developer’s  testing  and  the 
description  of  the  gaps  and  inconsistencies  in  the  Developer’s  testing  results  to  estimate  the  coverage  of 
the  Developer’s  testing. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  testing 
(Source:  Task  7.1.6) 

•  Description  of  the  gaps  and  inconsistencies  in  the  Developer’s  testing  results  (Source:  Task  7. 1 .7) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  coverage  of  the  Developer’s  testing  results 

7.1.9  Analyze  the  description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  testing  and  the 
estimated  coverage  of  the  Developer’s  testing  to  estimate  the  uncertainties  in  the  Developer’s  testing. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  testing 
(Source:  Task  7.1.6) 

•  Estimated  coverage  of  the  Developer’s  testing  (Source:  Task  7.1.8) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  uncertainties  in  the  Developer’s  testing  results 

7. 1. 10  Analyze  the  description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  testing,  the 
description  of  the  gaps  and  inconsistencies  in  the  Developer’s  testing  results,  the  estimated  coverage 
of,  and  uncertainties  in,  the  Developer’s  testing  results  and  the  acceptability  criteria  to  identify 
candidate  output  sampling  areas. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  testing 
(Source:  Task  7.1.6) 

•  Description  of  the  gaps  and  inconsistencies  in  the  Developer’s  testing  results  (Source:  Task  7. 1 .7) 

•  Estimated  coverage  of  the  Developer’s  testing  results  (Source:  Task  7. 1.8) 

•  Estimated  uncertainties  in  the  Developer’s  testing  results  (Source:  Task  7.1.9) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Candidate  output  sampling  areas  suggested  by  the  Developer’s  testing 


Activity  7.2  -  Plan  for  Results  Validation 

Activity  Description:  In  this  activity,  the  V&V  practitioner  plans  the  results  validation  effort.  This  can 
involve  applying  design  of  experiments  techniques  to  build  the  test  cases  used  to  sample  the  M&S’s  output. 

Activity  Tailoring  Guidance:  (Blank) 
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Activity  Notes: 

•  This  planning  activity  requires  solving  the  problem  of  describing  the  planning  process  in  a  way 
general  enough  to  admit  a  number  of  techniques,  including  SMEs. 

•  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified  in  the 
V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

7.2.1  Obtain  the  M&S  user  documentation  from  the  Developer. 

Information  Required: 

•  M&S  user  documentation  (Source:  Developer) 

Information  Produced: 

•  M&S  user  documentation 

7.2.2  Analyze  the  candidate  output  sampling  areas  from  the  prior  V&V  activities  and  the  Developer’s  testing 
to  produce  an  integrated  set  of  output  sampling  areas. 

Information  Required: 

•  Candidate  output  sampling  areas  suggested  by  the  historical  evidence  (Source:  Activity  2.4) 

•  Candidate  output  sampling  areas  implied  by  the  scenario  and  M&S  conceptual  model  V&V 
(Source:  Activity  3.4) 

•  Candidate  output  sampling  areas  implied  by  the  development  product  verification  evidence 
(Source:  Activity  5.3) 

•  Candidate  output  sampling  areas  implied  by  the  data  and  knowledge  set  V&V 
(Source:  Activity  6.4) 

•  Candidate  output  sampling  areas  suggested  by  the  Developer’s  testing  (Source:  Activity  7. 1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  set  of  output  sampling  areas 

7.2.3  Analyze  the  integrated  set  of  output  sampling  areas  and  the  acceptability  criteria  to  identify  the  M&S 
capabilities  and  limitations  that  will  be  tested. 

Information  Required: 

•  Integrated  set  of  output  sampling  areas  (Source:  Task  7.2.2) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

Information  Produced: 

•  M&S  capabilities  and  limitations  that  will  be  tested 

7.2.4  Analyze  the  M&S  capabilities  and  limitations  that  will  be  tested,  verified  and  validated  collected 
scenarios  and  the  acceptability  criteria  to  determine  test  scenarios  that  adequately  represent  the 
intended  use. 
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Information  Required: 

•  M&S  capabilities  and  limitations  that  will  be  tested  (Source:  Task  7.2.3) 

•  Verified  and  validated  collected  scenarios  (Source:  Activity  3.3) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Test  scenarios  that  represent  the  intended  use 

7.2.5  Analyze  the  M&S  user  documentation,  integrated  set  of  output  sampling  areas,  M&S  capabilities  that 
will  be  tested  and  test  scenarios  that  represent  the  intended  use  to  design  the  test  cases  for  results 
validation. 

Information  Required: 

•  M&S  user  documentation  (Source:  Task  7.2.1) 

•  Integrated  set  of  output  sampling  areas  (Source:  Task  7.2.2) 

•  M&S  capabilities  and  limitations  that  will  be  tested  (Source:  Task  7.2.3) 

•  Test  scenarios  that  represent  the  intended  use  (Source:  Task  7.2.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Test  cases  for  results  validation  testing 

7.2.6  Analyze  the  M&S  user  documentation,  test  scenarios  that  represent  the  intended  use,  the  test  cases  for 
results  validation  and  the  M&S  design  products  to  design  the  test  procedures  for  results  validation. 

Information  Required: 

•  M&S  user  documentation  (Source:  Task  7.2.1) 

•  Test  scenarios  that  represent  the  intended  use  (Source:  Task  7.2.4) 

•  Test  cases  for  results  validation  (Source:  Task  7.2.5) 

•  Verified  M&S  design  products  (Source:  Activity  4.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Test  procedures  for  results  validation  testing 

7.2.7  Prepare  the  results  validation  test  plan. 

Information  Required: 

•  Integrated  set  of  output  sampling  areas  (Source:  Task  7.2.2) 

•  M&S  capabilities  and  limitations  that  will  be  tested  (Source:  Task  7.2.3) 

•  Test  scenarios  that  represent  the  intended  use  (Source:  Task  7.2.4) 

•  Test  cases  for  results  validation  testing  (Source:  Task  7.2.5) 

•  Test  procedures  for  results  validation  testing  (Source:  Task  7.2.6) 

Information  Produced: 

•  Results  validation  test  plan 
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Activity  7.3  -  Collect  M&S  Output  for  Results  Validation 

Activity  Description:  In  this  activity,  the  V&V  practitioner  executes  the  results  validation  test  cases  and 
collects  the  output  needed  for  results  validation. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

7.3.1  Test  the  M&S  according  to  the  results  validation  plan. 

Information  Required: 

•  Results  validation  test  plan  (Source:  Activity  7.2) 

Information  Produced: 

•  M&S  output  from  V&V  testing 

7.3.2  Obtain  the  M&S  output  from  the  T&E  testing. 

Information  Required: 

•  M&S  output  from  T&E  testing  (Source:  T&E  Team) 

Information  Produced: 

•  M&S  output  from  T&E  testing 

7.3.3  Obtain  the  conditions  under  which  the  T&E  testing  was  performed. 

Information  Required: 

•  M&S  T&E  testing  conditions  (Source:  T&E  Team) 

Information  Produced: 

•  M&S  T&E  testing  conditions 

7.3.4  Characterize  the  conditions  under  which  the  M&S  results  were  collected. 

Information  Required: 

•  Results  validation  test  plan  (Source:  Activity  7.2) 

•  M&S  output  from  V&V  testing  (Source:  Task  7.3.1) 

Information  Produced: 

•  M&S  validation  testing  conditions 

7.3.5  Analyze  the  output  produced  by  the  results  validation  testing  and  the  validated  M&S  conceptual  model 
to  determine  the  M&S  capabilities  and  limitations  implied  by  that  output. 
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ORGANIZATION 


Information  Required: 

•  M&S  output  from  V&V  testing  (Source:  Task  7.3.1) 

•  M&S  output  from  T&E  testing  (Source:  Task  7.3.2) 

•  Verified  and  validated  M&S  design  products  (Source:  Activity  4.2) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Description  of  M&S  capabilities  and  limitations  implied  by  results  validation  testing 

7.3.6  Analyze  the  M&S  capabilities  and  limitations  implied  by  the  results  validation  testing  and  the 
acceptability  criteria  to  identify  any  gaps  and  inconsistencies  that  may  exist  in  those  testing  results. 

Information  Required: 

•  Description  of  M&S  capabilities  and  limitations  implied  by  results  validation  testing 
(Source:  Task  7.3.5) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Description  of  the  gaps  and  inconsistencies  in  the  results  validation  testing 

7.3.7  Analyze  the  descriptions  of  M&S  capabilities  and  limitations  and  the  gaps  and  inconsistencies  in  the 
results  validation  testing  and  the  acceptability  criteria  to  estimate  the  coverage  of  the  results  validation 
testing. 

Information  Required: 

•  Description  of  M&S  capabilities  and  limitations  implied  by  results  validation  testing 
(Source:  Task  7.3.5) 

•  Description  of  the  gaps  and  inconsistencies  in  the  results  validation  testing  (Source:  Task  7.3.6) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  coverage  of  the  results  validation  testing 

7.3.8  Analyze  the  description  of  M&S  capabilities  and  limitations  implied  by  and  the  estimated  coverage  of 
the  results  validation  testing  to  estimate  the  uncertainties  in  the  results  validation  testing. 

Information  Required: 

•  Description  of  M&S  capabilities  and  limitations  implied  by  results  validation  testing 
(Source:  Task  7.3.5) 

•  Estimated  coverage  of  the  results  validation  testing  (Source:  Task  7.3.7) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 
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Information  Produced: 

•  Estimated  uncertainties  in  the  results  validation  testing 

7.3.9  Repeat  until  the  results  validation  testing  has  produced  complete  and  confident  enough  results. 
Information  Required: 

•  Estimated  coverage  of  the  results  validation  testing  (Source:  Task  7.3.7) 

•  Estimated  uncertainties  in  the  results  validation  testing  (Source:  Task  7.3.8) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Completed  set  of  test  results 


Activity  7.4  -  Integrate  and  Employ  the  Validation  Testing  Results 

Activity  Description:  In  this  activity,  the  V&V  practitioner  infers  the  validity  of  the  M&S  from  the  collected 
M&S  output. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  Tasks  identify  the  V&V  Plan  as  input  because  they  apply  the  techniques  and  tools  identified 
in  the  V&V  Plan  to  perform  the  analyses. 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

7.4. 1  Analyze  the  results  from  the  Developer’s  testing  and  the  results  validation  testing  to  produce  an 
integrated  description  of  the  M&S  capabilities  and  limitations  from  results  validation. 

Information  Required: 

•  Description  of  the  M&S  capabilities  and  limitations  from  the  Developer’s  testing 
(Source:  Activity  7.1) 

•  Description  of  M&S  capabilities  and  limitations  implied  by  results  validation  testing 
(Source:  Activity  7.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  results  validation 

7.4.2  Analyze  the  gaps  and  inconsistencies  identified  by  the  Developer’s  testing  and  the  results  validation 
testing  using  the  integrated  description  of  the  M&S  capabilities  and  limitations  from  the  M&S  testing 
to  reconcile  them  and  produce  a  description  of  the  unresolved  gaps  and  inconsistencies  in  the  results 
validation  evidence. 
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Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  results  validation 
(Source:  Task  7.4.1) 

•  Description  of  the  gaps  and  inconsistencies  in  the  Developer’s  testing  results  (Source:  Activity  7.1) 

•  Description  of  the  gaps  and  inconsistencies  in  the  results  validation  testing  (Source:  Activity  7.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  results  validation  evidence 

7.4.3  Analyze  the  integrated  description  of  M&S  capabilities  and  limitations,  the  description  of  the  unresolved 
gaps  and  inconsistencies  in  the  results  validation  evidence  and  the  acceptability  criteria  to  estimate  the 
coverage  of  the  results  validation  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  results  validation 
(Source:  Task  7.4.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  results  validation  evidence 
(Source:  Task  7.4.2) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Required: 

•  Estimated  coverage  of  the  results  validation  evidence 

7.4.4  Analyze  the  integrated  description  of  M&S  capabilities  and  limitations  and  the  estimated  coverage  of 
the  results  validation  evidence  to  estimate  the  uncertainties  in  the  integrated  results  validation  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  results  validation 
(Source:  Task  7.4.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  results  validation  evidence 
(Source:  Task  7.4.2) 

•  Estimated  coverage  of  the  results  validation  evidence  (Source:  Task  7.4.3) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  uncertainties  in  the  results  validation  evidence 

7.4.5  Analyze  the  acceptability  criteria,  the  integrated  description  of  M&S  capabilities  and  limitations  from 
the  results  validation  evidence  and  the  estimated  uncertainties  in  the  results  validation  evidence  to 
identify  the  factors  that  may  constrain  use. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  results  validation 
(Source:  Task  7.4.1) 

•  Estimated  uncertainties  in  the  results  validation  evidence  (Source:  Task  7.4.4) 

•  V&V  Plan  (Source:  Activity  1.6) 
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Information  Produced: 

•  Factors  that  may  constrain  use  implied  by  the  results  validation  evidence 

7.4.6  Prepare  the  source  reference  list  supporting  results  validation  and  resolve  any  citation  problems. 
Information  Required: 

•  Developer’s  test  plan  (Source:  Activity  7.1) 

•  Developer’s  testing  results  (Source:  Activity  7.1) 

•  M&S  user  documentation  (Source:  Activity  7.2) 

•  M&S  output  from  T&E  testing  (Source:  Activity  7.3) 

•  M&S  T&E  testing  conditions  (Source:  Activity  7.3) 

Information  Produced: 

•  Citations  of  information  supporting  results  validation 

7.4.7  Update  the  requirements  traceability  matrix  with  the  results  validation  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  results  validation 
(Source:  Task  7.4.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  results  validation  evidence 
(Source:  Task  7.4.2) 

•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Requirements  traceability  matrix  updated  with  the  results  validation  evidence 
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A1.9  PHASE  8  -  INTEGRATE  THE  V&V  EVIDENCE 
Activity  8.1  -  Integrate  the  V&V  Evidence 

Activity  Description:  This  activity  integrates  the  V&V  evidence  into  a  single  consistent  description. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  (Blank) 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

8.1.1  Analyze  the  collected  evidence  produced  by  the  V&V  activities  to  generate  an  integrated  description 

of  the  M&S  capabilities  and  limitations. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  historical  evidence 
(Source:  Activity  2.4) 

•  Integrated  description  of  the  M&S  capabilities  that  the  scenarios  and  M&S  conceptual  model  can 
support  (Source:  Activity  3.4) 

•  Integrated  description  of  the  M&S  limitations  implied  by  the  scenario  and  M&S  conceptual 
model  V&V  (Source:  Activity  3.4) 

•  Needed  M&S  capabilities  that  the  M&S  executable  could  support  (Source:  Activity  5.2) 

•  M&S  limitations  implied  by  the  development  product  verification  evidence  (Source:  Activity  5.2) 

•  Integrated  description  of  the  M&S  capabilities  that  the  data  and  knowledge  sets  can  support 
(Source:  Activity  6.4) 

•  Integrated  description  of  the  M&S  limitations  implied  by  the  data  and  knowledge  V&V 
(Source:  Activity  6.4) 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  results  validation 
(Source:  Activity  7.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence 

8.1.2  Analyze  the  gaps  and  inconsistencies  identified  by  the  prior  V&V  activities  using  the  integrated 

description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence  to  reconcile  them 

and  produce  a  description  of  the  unresolved  gaps  and  inconsistencies  in  the  V&V  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence 
(Source:  Task  8.1.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  historical  evidence 
(Source:  Activity  2.4) 

•  Integrated  description  of  the  information  gaps  and  inconsistencies  in  the  evidence  from  the 
scenario  and  M&S  conceptual  model  V&V  (Source:  Activity  3.4) 
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•  Gaps  and  inconsistencies  in  the  evidence  from  the  Developer’s  verification  products 
(Source:  Activity  4.1) 

•  Integrated  description  of  the  inconsistencies  discovered  in  the  M&S  development  products 
(Source:  Activity  4.6) 

•  Information  gaps  in  the  development  product  verification  evidence  (Source:  Activity  5.2) 

•  Integrated  description  of  the  information  gaps  implied  by  the  data  and  knowledge  set  V&V 
(Source:  Activity  6.4) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  results  validation  evidence 
(Source:  Activity  7.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  collected  V&V  evidence 

8. 1 .3  Analyze  the  integrated  description  of  M&S  capabilities  and  limitations,  the  description  of  the  unresolved 
gaps  and  inconsistencies  in  the  integrated  V&V  evidence  and  the  acceptability  criteria  to  estimate  the 
coverage  of  the  integrated  V&V  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence 
(Source:  Task  8.1.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  collected  V&V  evidence 
Source:  Task  8.1.2) 

•  Estimated  coverage  of  the  historical  evidence  (Source:  Activity  2.4) 

•  Estimated  coverage  of  the  scenario  and  M&S  conceptual  model  V&V  evidence 

(Source:  Activity  3.4) 

•  Estimated  coverage  of  the  Developer’s  verification  products  (Source:  Activity  4.1) 

•  Estimated  coverage  of  the  development  product  verification  evidence  (Source:  Activity  5.2) 

•  Estimated  coverage  of  the  data  and  knowledge  set  V&V  evidence  (Source:  Activity  6.4) 

•  Estimated  coverage  of  the  results  validation  evidence  (Source:  Activity  7.4) 

•  Acceptability  criteria  defining  needed  functionality  (Source:  Activity  1.1) 

•  Acceptability  criteria  defining  needed  quality  (Source:  Activity  1.1) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  coverage  of  the  collected  V&V  evidence 

8.1.4  Analyze  the  integrated  description  of  M&S  capabilities  and  limitations  and  the  estimated  coverage  of 
the  V&V  evidence  to  estimate  the  uncertainties  in  the  integrated  V&V  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence 
(Source:  Task  8.1.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  collected  V&V  evidence 
(Source:  Task  8.1.2) 

•  Estimated  coverage  of  the  collected  V&V  evidence  (Source:  Task  8.1.3) 

•  Estimated  uncertainties  in  the  historical  evidence  (Source:  Activity  2.4) 

•  Estimated  uncertainties  in  the  M&S  conceptual  model  V&V  evidence  (Source:  Activity  3.4) 

•  Estimated  uncertainties  in  the  development  product  verification  evidence  (Source:  Activity  5.2) 
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•  Estimated  uncertainties  in  the  data  and  knowledge  set  V&V  evidence  (Source:  Activity  6.4) 

•  Estimated  uncertainties  in  the  results  validation  evidence  (Source:  Activity  7.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Estimated  uncertainties  in  the  collected  V&V  evidence 

8.1.5  Analyze  the  acceptability  criteria,  the  integrated  description  of  M&S  capabilities  and  limitations  from 
the  V&V  evidence  and  the  estimated  uncertainties  in  the  V&V  evidence  to  identify  the  factors  that 
may  constrain  use. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence 
(Source:  Task  8.1.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  collected  V&V  evidence 
(Source:  Task  8.1.2) 

•  Estimated  uncertainties  in  the  collected  V&V  evidence  (Source:  Task  8.1.4) 

•  Factors  that  may  constrain  use  implied  by  the  historical  evidence  (Source:  Activity  2.4) 

•  Factors  that  may  constrain  use  implied  by  the  scenario  and  M&S  conceptual  model  V&V 

(Source:  Activity  3.4) 

•  Factors  that  may  constrain  use  implied  by  the  development  product  verification  evidence 
(Source:  Activity  5.3) 

•  Factors  that  may  constrain  use  implied  by  the  data  and  knowledge  set  V&V  (Source:  Activity  6.4) 

•  Factors  that  may  constrain  use  implied  by  the  results  validation  evidence  (Source:  Activity  7.4) 

•  V&V  Plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Factors  that  may  constrain  use  implied  by  the  collected  V&V  evidence 

8.1.6  Prepare,  from  the  individual  citation  lists,  an  integrated  list  of  citations  of  relevant  and  applicable 
information  and  resolve  any  citation  problems. 

Information  Required: 

•  Citations  of  information  used  for  V&V  Planning  (Source:  Activity  1.5) 

•  Integrated  citations  supporting  the  compilation  and  analysis  of  the  historical  evidence 
(Source:  Activity  2.4) 

•  Integrated  citations  supporting  the  compilation  and  analysis  of  the  M&S  conceptual  model  and 
collected  scenarios  (Source:  Activity  3.4) 

•  Integrated  citations  supporting  the  development  of  the  verification  evidence  (Source:  Activity  4.6) 

•  Data  and  knowledge  set  pedigree  citation  list  (Source:  Activity  6.1) 

•  Citations  of  information  supporting  results  validation  (Source:  Activity  7.4) 

Information  Produced: 

•  Integrated  citations  supporting  the  compilation  and  analysis  of  the  collected  V&V  evidence 

8.1.7  Update  the  requirements  traceability  matrix  with  the  collected  V&V  evidence. 
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Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence 
(Source:  Task  8.1.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  collected  V&V  evidence 
(Source:  Task  8.1.2) 

•  Requirements  traceability  matrix  (Source:  Activity  1.6) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  Requirements  traceability  matrix  updated  with  the  collected  V&V  evidence 


Activity  8.2  -  Derive  the  Conclusions  from  the  Integrated  V&V  Evidence 

Activity  Description:  This  activity  develops  the  conclusions,  recommendations  and  lessons  learned  from  the 
integrated  V&V  results  and  the  V&V  Plan.  It  also  develops  the  presentations  of  the  V&V  results  (e.g.,  graphics, 
tables). 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  (Blank) 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

8.2.1  Develop  presentations  of  the  collected  V&V  results  (e.g.,  graphics,  tables). 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence 
(Source:  Activity  8.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  collected  V&V  evidence 
(Source:  Activity  8.1) 

•  Estimated  coverage  of  the  collected  V&V  evidence  (Source:  Activity  8.1) 

•  Estimated  uncertainties  in  the  collected  V&V  evidence  (Source:  Activity  8.1) 

Information  Produced: 

•  Presentations  of  the  collected  V&V  results 

8.2.2  Prepare  a  summary  of  the  collected  V&V  results. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence 
(Source:  Activity  8.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  collected  V&V  evidence 
(Source:  Activity  8.1) 

•  Estimated  coverage  of  the  collected  V&V  evidence  (Source:  Activity  8.1) 

•  Estimated  uncertainties  in  the  collected  V&V  evidence  (Source:  Activity  8.1) 
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Information  Produced: 

•  Summary  of  the  collected  V&V  results 

8.2.3  Prepare  a  summary  of  the  lessons  learned  from  the  V&V  effort. 

Information  Required: 

•  Collected  lessons  learned  from  exercising  the  V&V  process  (Source:  Activity  1.6) 

Information  Produced: 

•  Summary  of  the  lessons  learned  from  the  V&V  effort 

8.2.4  Analyze  the  summary  of  the  collected  V&V  results,  summary  of  the  lessons  learned  from  the  V&V 
effort  and  M&S  requirements  to  develop  the  V&V  recommendations. 

Information  Required: 

•  Summary  of  the  collected  V&V  results  (Source:  Task  8.2.2) 

•  Summary  of  the  lessons  learned  from  the  V&V  effort  (Source:  Task  8.2.3) 

•  M&S  requirements  (Source:  Activity  1.1) 

Information  Produced: 

•  V&V  recommendations 


Activity  8.3  -  Prepare  the  V&V  Report 

Activity  Description:  This  activity  prepares  the  V&V  Report  from  the  integrated  V&V  evidence. 

Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  (Blank) 

Activity  Assumptions:  (Blank) 

Task  Descriptions: 

8.3.1  Analyze  the  V&V  Plan  to  develop  the  problem  statement  for  the  V&V  effort. 

Information  Required: 

•  V&V  plan  (Source:  Activity  1.5) 

Information  Produced: 

•  V&V  effort  problem  statement 

8.3.2  Prepare  a  summary  of  the  M&S. 

Information  Required: 

•  V&V  plan  (Source:  Activity  1.5) 

Information  Produced: 

•  Summary  of  the  M&S 
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8.3.3  Prepare  a  summary  of  the  techniques  and  tools  applied  to  execute  the  V&V  tasks. 

Information  Required: 

•  V&V  plan  (Source:  Activity  1.6) 

Information  Produced: 

•  Summary  of  techniques  and  tools  applied  in  the  V&V  effort 

8.3.4  Prepare  a  summary  of  the  conditions  under  which  the  M&S  was  tested. 

Information  Required: 

•  Documentation  from  prior  testing  efforts  (Source:  Activity  2.2) 

•  Relevant  and  applicable  Developer’s  verification  products  (Source:  Activity  4.1) 

•  M&S  executable  verification  testing  conditions  (Source:  Activity  4.4) 

•  M&S  validation  testing  conditions  (Source:  Activity  7.3) 

Information  Produced: 

•  Summary  of  the  conditions  under  which  the  M&S  was  tested 

8.3.5  Describe  the  integrated  evidence. 

Information  Required: 

•  Integrated  description  of  the  M&S  capabilities  and  limitations  from  the  collected  V&V  evidence 
(Source:  Activity  8.1) 

•  Description  of  the  unresolved  gaps  and  inconsistencies  in  the  collected  V&V  evidence 
(Source:  Activity  8.1) 

•  Estimated  coverage  of  the  collected  V&V  evidence  (Source:  Activity  8.1) 

•  Estimated  uncertainties  in  the  collected  V&V  evidence  (Source:  Activity  8.1) 

•  Factors  that  may  constrain  use  implied  by  the  collected  V&V  evidence  (Source:  Activity  8.1) 

•  Integrated  citations  supporting  the  compilation  and  analysis  of  the  collected  V&V  evidence 
(Source:  Activity  8.1) 

Information  Produced: 

•  Description  of  the  integrated  V&V  evidence 

8.3.6  Analyze  the  problem  statement  and  the  description  of  integrated  evidence  to  develop  the  V&V 
recommendations. 

Information  Required: 

•  V&V  effort  problem  statement  (Source:  Task  8.3.1) 

•  Description  of  the  integrated  V&V  evidence  (Source:  Task  8.3.5) 

Information  Produced: 

•  V&V  recommendations 

8.3.7  Describe  the  key  participants  in  and  contributors  to  the  V&V  effort. 

Information  Required: 

•  Key  participants’  background  and  expertise  information 
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Information  Produced: 

•  Key  participants’  description 

8.3.8  Obtain  the  resources  expended  from  the  V&V  project  accounting  team. 

Information  Required: 

•  V&V  expenditure  information  (Source:  V&V  Project  Accounting  Team) 
Information  Produced: 

•  Resources  expended  in  executing  the  V&V  effort 

8.3.9  Describe  the  resources  expended  in  executing  the  V&V  effort. 

Information  Required: 

•  Resources  expended  in  executing  the  V&V  effort  (Source:  Task  8.3.8) 
Information  Produced: 

•  Description  of  the  resources  expended  for  the  V&V  effort 

8.3.10  Summarize  the  lessons  learned  from  the  V&V  effort. 

Information  Required: 

•  Collected  lessons  learned  from  exercising  the  V&V  process  (Source:  Activity  1.6) 

Information  Produced: 

•  Lessons  learned  from  the  V&V  effort 

8.3.11  Prepare  the  V&V  report  executive  summary. 

Information  Required: 

•  V&V  effort  problem  statement  (Source:  Task  8.3.1) 

•  Summary  of  the  M&S  (Source:  Task  8.3.2) 

•  Description  of  the  integrated  V&V  evidence  (Source:  Task  8.3.5) 

•  V&V  recommendations  (Source:  Task  8.3.6) 

•  Lessons  learned  from  the  V&V  effort  (Source:  Task  8.3.10) 

Information  Produced: 

•  V&V  report  executive  summary 

8.3.12  Assemble  the  V&V  Report. 

Information  Required: 

•  V&V  effort  problem  statement  (Source:  Task  8.3.1) 

•  Summary  of  the  M&S  (Source:  Task  8.3.2) 

•  Summary  of  techniques  and  tools  applied  in  the  V&V  effort  (Source:  Task  8.3.3) 

•  Summary  of  the  conditions  under  which  the  M&S  was  tested  (Source:  Task  8.3.4) 

•  Description  of  the  integrated  V&V  evidence  (Source:  Task  8.3.5) 

•  V&V  recommendations  (Source:  Task  8.3.6) 
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•  Key  participants  description  (Source:  Task  8.3.7) 

•  Description  of  the  resources  expended  for  the  V&V  effort  (Source:  Task  8.3.9) 

•  Lessons  learned  from  the  V&V  effort  (Source:  Task  8.3.10) 

•  V&V  report  executive  summary  (Source:  Task  8.3. 1 1) 

Information  Produced: 

•  Draft  V&V  Report 

8.3.13  Analyze  the  draft  V&V  report  for  internal  consistency  and  correct  any  inconsistencies. 

Information  Required: 

•  Draft  V&V  report  (Source:  Task  8.3.12) 

Information  Produced: 

•  Corrected  draft  V&V  report 

8.3.14  Circulate  the  corrected  draft  V&V  report  for  review  and  collect  comments  and  recommendations  for 
change. 

Information  Required: 

•  Corrected  draft  V&V  report  (Source:  Task  8.3. 13) 

Information  Produced: 

•  Comments  and  recommendations  for  change  received  on  draft  V&V  report 

8.3.15  Modify  the  corrected  draft  V&V  report  from  the  comments  and  recommendations  for  change  received. 
Information  Required: 

•  Corrected  draft  V&V  report  (Source:  Task  8.3. 13) 

•  Comments  and  recommendations  for  change  received  on  draft  V&V  report  (Source:  Task  8.3.14) 

Information  Produced: 

•  Modified  V&V  report 

8.3.16  Analyze  the  modified  V&V  report  for  internal  consistency,  correct  any  inconsistencies  and  prepare 
the  final  V&V  report. 

Information  Required: 

•  Modified  V&V  report  (Source:  Task  8.3.15) 

Information  Produced: 

•  Final  V&V  Report 


Activity  8.4  -  Support  any  Archival  of  the  V&V  Products 

Activity  Description:  This  activity  supports  any  needed  archival  of  the  V&V  products. 
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Activity  Tailoring  Guidance:  (Blank) 

Activity  Notes:  (Blank) 

Activity  Assumptions:  The  Users  will  decide  whether  and  where  to  archive  the  V&V  products. 

Task  Descriptions: 

8.4.1  Obtain  the  locations  for  the  archived  V&V  products. 

Information  Required: 

•  Selected  archive  locations  (Source:  User) 

Information  Produced: 

•  Selected  archive  locations 

8.4.2  Obtain  the  points  of  contact  for  the  selected  archive  locations. 

Information  Required: 

•  Selected  archive  locations  (Source:  Task  8.4.1) 

Information  Produced: 

•  Archive  location  points  of  contact 

8.4.3  Obtain  the  choices  of  resource  catalogs  within  which  to  publish  the  V&V  product  descriptions. 
Information  Required: 

•  Applicable  resource  catalogs  (Source:  User) 

Information  Produced: 

•  Applicable  resource  catalogs’  information  requirements 

8.4.4  Arrange  for  archival  of  V&V  products  through  the  archive  points  of  contact. 

Information  Required: 

•  Archive  location  points  of  contact  (Source:  Task  8.4.2) 

Information  Produced: 

•  Requirements  for  V&V  product  archive 

8.4.5  Determine  required  formats  and  media  for  archived  material. 

Information  Required: 

•  Requirements  for  V&V  product  archive  (Source:  Task  8.4.4) 

Information  Produced: 

•  Archival  format  requirements 

•  Archival  media  requirements 
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8.4.6  Translate  the  V&V  products  into  the  required  format. 
Information  Required: 

•  Archival  format  requirements  (Source:  Task  8.4.5) 

•  V&V  Plan  (Source:  Activity  1.5) 

•  V&V  Report  (Source:  Activity  8.3) 

Information  Produced: 

•  Formatted  V&V  Plan 

•  Formatted  V&V  Report 

8.4.7  Transfer  the  V&V  products  onto  the  required  media. 
Information  Required: 

•  Archival  media  requirements  (Source:  Task  8.4.5) 

•  Formatted  V&V  Plan  (Source:  Task  8.4.6) 

•  Formatted  V&V  Report  (Source:  Task  8.4.6) 

Information  Produced: 

•  Transferred  V&V  Plan 

•  Transferred  V&V  Report 

8.4.8  Deliver  the  archival  V&V  products  to  the  archival  location. 
Information  Required: 

•  Selected  archive  locations  (Source:  Task  8.4.1) 

•  Transferred  V&V  Plan  (Source:  Task  8.4.7) 

•  Transferred  V&V  Report  (Source:  Task  8.4.7) 

Information  Produced: 

•  Delivered  V&V  Plan 

•  Delivered  V&V  Report 

8.4.9  Determine  the  requirements  for  resource  catalog  descriptions. 
Information  Required: 

•  Applicable  resource  catalogs  (Source:  Task  8.4.3) 

Information  Produced: 

•  Resource  description  requirements 

8.4. 10  Prepare  descriptions  of  the  archived  V&V  products. 
Information  Required: 

•  Resource  description  requirements  (Source:  Task  8.4.9) 

•  V&V  Plan  (Source:  Activity  1.1) 

•  V&V  Report  (Source:  Activity  8.3) 
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Information  Produced: 

•  V&V  Plan  description 

•  V&V  Report  description 

8.4. 1 1  Publish  the  archived  product  descriptions  in  the  applicable  resource  catalogs. 
Information  Required: 

•  Applicable  resource  catalogs  (Source:  Task  8.4.3) 

•  V&V  Plan  description  (Source:  Task  8.4. 10) 

•  V&V  Report  description  (Source:  Task  8.4. 10) 

Information  Produced: 

•  Published  V&V  product  descriptions 
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ABSTRACT 

Using  risk  as  an  effective  tailoring  mechanism  for  Verification,  Validation  and  Accreditation  (VV&A) 
processes  depends  on  understanding  the  factors  that  drive  the  risk  as  well  as  the  VV&A  tasks  and  techniques 
that  can  and  should  be  implemented  to  address  the  use  risks.  This  paper  describes  one  perspective  of  the 
activities  and  tasks  that  constitute  the  VV&A  processes  for  simulations  upon  which  to  found  risk-based  VV&A 
tailoring.  This  perspective  describes  a  VV&A  process  model  that  builds  from  the  experience  gained  in 
developing  the  Validation  Process  Maturity  Model  (VPMM)  and  IEEE  Std  1516.4-2007,  IEEE  Recommended 
Practice  for  Verification,  Validation  and  Accreditation  of  a  Federation.  This  model  generalizes  the  guidance 
provided  in  IEEE  Std  1516.4-2007  to  apply  to  simulations  beyond  just  federations,  and  extends  that  guidance 
to  consider  the  broad  range  of  possible  sources  of  validation  evidence.  It  also  establishes  the  basis  from 
which  to  extend  the  applicability  of  the  VPMM  to  a  more  practical  extent  of  circumstances.  This  model  defines 
the  accreditation  process  in  terms  of  the  activities  for  planning  the  accreditation  effort,  characterizing  the 
simulation ’s  use,  constructing  the  validation  referent  and  developing  the  acceptance  recommendations.  It  defines 
the  V&V  process  in  terms  of  the  activities  for  planning  the  V&V  effort,  applying  historical  information, 
validating  the  simulation  conceptual  model,  leveraging  the  developer’s  verification  products,  performing 
supplemental  verification,  validating  the  simulation  results  and  integrating  the  V&V  results  into  a  coherent 
set  of  validation  evidence.  This  model  of  the  VV&A  processes  defines  a  superset  of  the  tasks  that  may  then  be 
tailored  to  best  suit  the  limits  of  acceptable  use  risk  and  optimize  the  resources  available  for  W&A.  This  paper 
describes  the  components  of  the  high  level  VV&A  activities  and  the  products  that  they  produce. 

Keywords:  validation,  verification,  accreditation,  tailoring. 


1.0  INTRODUCTION 

One  approach  to  tailoring  Verification,  Validation  and  Accreditation  (VV&A)  processes  begins  with  a  model 
that  defines  the  spectrum  of  activities  and  tasks  that  could  be  done  when  performing  VV&A  then  selecting 
and  assembling  the  components  of  that  model  to  best  match  the  risk  and  resource  constraints  of  the  simulation 
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effort  while  still  adhering  to  the  policies,  standards  and  guidelines  that  may  be  relevant  to  the  intended  use  [1], 
This  paper  presents  such  a  model  of  VV&A  processes  that  practitioners  can  tailor  to  meet  the  needs  and 
constraints  of  their  particular  situations.  This  model  can  help  the  VV&A  community  realize  the  goal  of 
tailorability  that  several  sources  have  cited  [l]-[3]. 

Four  assumptions  underlie  the  construction  of  this  VV&A  process  model: 

•  This  general  model  represents  the  breadth  of  the  VV&A  process  from  user  requirements  to  the 
acceptance  decision  for  an  intended  use. 

•  Accreditation  refers  to  the  process  for  arriving  at  a  decision  to  accept  or  accredit  a  simulation  for  an 
intended  use. 

•  The  primary  purpose  for  performing  Verification  and  Validation  (V&V)  is  to  collect  the  evidence 
needed  to  support  an  acceptance  decision  for  an  intended  use. 

•  The  simulation  users  or  their  representatives  are  responsible  for  ensuring  the  completeness  and 
correctness  of  any  statements  of  user  needs  or  simulation  requirements. 


The  model  in  this  paper  integrates  several  process  models  described  in  the  VV&A  literature  with  recent 
experiences  in  performing  VV&A  to  support  actual  simulation  programs.  The  result  is  a  superset  of  the  possible 
VV&A  activities  and  tasks  and  the  context  in  which  those  components  can  be  assembled  into  a  working 
VV&A  process.  This  model  is  described  in  two  parts:  an  accreditation  process  model  that  encompasses  the 
V&V  process  as  a  part  and  the  V&V  process  model  that  produces  the  validation  evidence  needed  by  the 
accreditation  process. 


2.0  ACCREDITATION  PROCESS  MODEL 

The  accreditation  process  takes  the  accreditation  needs,  user  needs  and  referent  source  information  as  input  and 
produces  the  acceptance  recommendations.  Figure  A2-1  illustrates  the  top  level  of  this  proposed  accreditation 
process  model. 
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Figure  A2-1:  Top  Level  Diagram  of  the  VV&A  Processes  and  Their  Products. 


This  model  defines  the  accreditation  process  in  terms  of  five  activities: 

•  Plan  the  accreditation  effort. 

•  Characterize  the  simulation’s  use. 

•  Construct  the  validation  referent. 

•  Collect  the  validation  evidence. 

•  Develop  the  acceptance  recommendations. 

The  sections  below  describe  each  of  these  activities. 


2.1  Plan  the  Accreditation  Effort 

Several  authors  have  included  accreditation  planning  as  a  critical  part  of  their  W&A  process  models  [l],[4]-[8]. 
Accreditation  planning  consists  of  six  tasks: 

•  Assess  the  accreditation  needs. 

•  Develop  the  accreditation  approach. 

•  Build  the  accreditation  schedule. 

•  Estimate  the  accreditation  costs. 

•  Prepare  the  accreditation  plan. 

•  Execute  and  evolve  the  accreditation  plan. 

In  assessing  the  accreditation  needs,  the  accreditation  agent  works  with  the  program  management  to  define  the 
scope  of  the  accreditation  effort.  This  knowledge  enables  the  accreditation  agent  to  develop  the  accreditation 


RTO-TR-MSG-054 


A2  -3 


APPENDIX  2  -  DECOMPOSING  THE 

VV&A  PROCESSES  TO  SUPPORT  THEIR  TAILORING 


ORGANIZATION 


approach,  schedule  the  accreditation  tasks,  estimate  the  costs  of  performing  those  tasks  and  capture  this 
information  in  the  accreditation  plan.  The  accreditation  agent  executes  and  evolves  the  accreditation  plan 
throughout  the  accreditation  effort.  This  includes  monitoring  the  accreditation  and  V&V  efforts  and  adjusting 
the  plans  to  better  reflect  any  new  information  gained  while  executing  the  accreditation  or  V&V  processes. 

The  task  of  developing  the  accreditation  approach  may  include  defining  and  prioritizing  the  application 
requirements  and  defining  the  needed  level  of  objectivity  as  described  in  Reference  [5]. 

As  is  true  throughout  this  paper,  the  order  of  planning  tasks  given  above  does  not  necessarily  imply  either 
their  order  of  execution  or  that  there  is  no  iteration  involved  within  this  activity  or  between  the  activities  in  the 
accreditation  process.  In  some  cases,  the  accreditation  agent  can  only  formulate  a  rough  plan  that  is  then  revised 
throughout  its  execution. 

2.2  Characterize  the  Simulation’s  Use 

Characterizing  the  simulation’s  use  includes  several  tasks  that  have  been  traditionally  associated  with  the 
accreditation  process  [  1  ] ,  [4] ,  [5  ] ,  [9] : 

•  Collect  the  user  needs. 

•  Verify  the  requirements/objectives. 

•  Document  the  intended  use. 

•  Assess  the  use  risks. 

•  Define  the  conditions  of  expected  use. 

•  Develop  the  acceptability  criteria. 

•  Specify  the  accreditation  information  requirements. 

The  first  task  collects  and  integrates  the  available  documented  requirements  information  (e.g.,  operational 
requirements  document,  capabilities  development  description)  as  well  as  elicits  needs  from  the  users  or  their 
representatives  in  order  to  develop  as  complete  a  picture  of  the  user  needs  as  possible.  Several  authors  have 
suggested  verifying  the  consistency  of  any  user  requirements  or  objectives  as  well  [1  ],[4],[9].  However,  this 
process  purposely  avoids  the  notion  of  validating  the  users’  needs  or  requirements  as  suggested  in  Reference 
[10]  because  the  authors  feel  that  this  intrudes  sharply  into  the  users’  domain.  As  assumed  above,  the  users  or 
their  representatives  need  to  ensure  the  completeness  and  correctness  of  their  requirements.  The  VV&A  team 
can  only  check  the  requirements  for  consistency  if  desired.  The  consistency  checking  process  may  uncover 
problems  in  the  requirements  statements  but  the  users  or  their  representatives  should  resolve  those  problems. 
Similarly,  the  VV&A  team  cannot  tailor  the  application  requirements  as  suggested  in  Reference  [5]. 

Knowledge  of  the  user  needs  enables  documenting  the  intended  use,  usually  a  brief  statement  of  how  the  users 
will  apply  the  simulation  results.  This  task  commonly  requires  considerable  iteration  with  the  users  or  their 
representatives  and,  possibly,  other  interested  parties  (e.g.,  developer).  Iteration  is  also  needed  in  assessing  the 
use  risks.  This  task  includes  identifying  the  risk  areas  and  defining  the  impact  levels  as  described  in  Reference 
[5],  The  resolution  of  the  use  risk  assessment  depends  upon  the  scope  of  the  accreditation  effort  and  the 
amount  of  information  available  at  the  time.  In  some  cases,  the  risk  assessment  may  only  involve  prioritizing 
the  user  needs.  In  other  cases,  the  use  risks  can  be  explicitly  described  subjectively  or  quantitatively  through 
such  techniques  as  fault  tree  analysis  [11].  The  accreditation  agent  should  understand  the  risks  of  using  a 
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simulation  as  well  as  the  simulation’s  capabilities  and  limitations  [12].  Failure  to  do  this  will  impact  the 
subsequent  accreditation  and  V&V  activities  and  tasks  [13]. 

The  acceptability  criteria  specify  the  thresholds  of  functionality  and  error  that  a  simulation  needs  to  meet  or 
exceed  in  order  to  be  unconditionally  accredited  for  an  intended  use  [13].  Acceptability  criteria  may  be  derived 
entirely  from  user  needs  documentation  (i.e.,  user  requirements)  [13],  through  an  iterative  review  and  interview 
process  [12]  or  some  combination  of  those  techniques.  If  sufficient  information  is  available,  the  acceptability 
criteria  may  be  prioritized  according  to  risk  [12].  The  assessment  of  tolerable  use  risks  can  contribute  to  this 
prioritization.  Reference  [13]  discusses  the  derivation  and  properties  of  acceptability  criteria  in  detail. 
The  conditions  of  expected  use  complement  the  acceptability  criteria  by  defining  the  properties  of  the  inputs  that 
the  user  expects  to  provide  to  the  simulation  and  the  outputs  that  they  desire  the  simulation  to  produce.  These 
conditions  define  the  bounds  of  V&V  testing  [2],  the  domain  of  applicability  of  the  acceptance  recommendations 
[3],  [14]  and,  thus,  of  the  intended  use  [2], [3].  Like  the  acceptability  criteria,  the  conditions  of  expected  use  need 
to  be  derived  from  the  user  needs  documentation  and  interactions  with  the  users  or  their  representatives. 

Finally,  the  accreditation  agent  assembles  the  assessment  of  use  risks,  conditions  of  expected  use  and  acceptability 
criteria  into  the  accreditation  information  requirements  that,  together  with  the  accreditation  plan,  guide  the 
V&V  effort.  In  many  cases,  the  accreditation  agent  may  need  to  iterate  between  this  activity  and  accreditation 
planning. 

2.3  Construct  the  Validation  Referent 

The  validation  referent  is  the  best  available  knowledge  about  the  things  being  simulated  [15], [16]  and  establishes 
the  standard  against  which  to  measure  simulation  error.  Constructing  the  validation  referent  involves  three  tasks: 

•  Identify  practical  referent  information  sources. 

•  Collect  the  applicable  referent  information. 

•  Assemble  the  validation  referent. 

The  sources  for  validation  referents  exist  in  many  forms,  ranging  from  subjective  and  qualitative  descriptions 
to  objective  and  quantitative  descriptions  [4]: 

•  Data  from  controlled  experiments  describing  the  functionality  and  performance  of  a  system  or 
phenomenon  under  well-known  conditions. 

•  Empirical  data  from  observations  of  the  behavior  of  a  system  or  phenomenon  under  conditions  ranging 
from  unknown  to  well-characterized. 

•  Experience,  knowledge  and  intuition  of  subject-matter  experts. 

•  Mathematical  models  of  the  behavior  of  a  system  or  phenomenon  that  have  been  validated  against 
experimental  or  empirical  data. 

•  Other  simulations  that  have  established  credibility  with  the  users  for  their  particular  intended  uses. 

•  Combinations  of  the  types  described  above. 

The  accreditation  agent  collects  the  information  from  these  referent  sources  and  assembles  that  information 
into  a  single  consistent  validation  referent  for  the  intended  use.  It  is  important  to  understand  the  uncertainties 
associated  with  any  referent  because  those  uncertainties  add  to  those  associated  with  the  validation  evidence. 
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In  addition,  validation  referents  need  to  be  credible  to  a  simulation’s  users  since  they  will  be  impacted  by  the 
simulation’s  deviation  from  the  behavior  of  the  thing  being  simulated. 

2.4  Collect  the  Validation  Evidence 

The  accreditation  process  and  the  acceptance  recommendations  both  depend  upon  the  evidence  that  the  V&V 
process  produces  [2],  [5].  The  collection  of  this  evidence  involves  eight  V&V  activities: 

•  Plan  the  V&V  effort. 

•  Apply  relevant  historical  information. 

•  Verify  and  validate  the  simulation  conceptual  model. 

•  Perform  supplementary  verification  on  the  development  products. 

•  Leverage  the  verification  products. 

•  Verify  and  validate  the  data  and  knowledge  sets. 

•  Validate  the  simulation  results. 

•  Integrate  the  validation  evidence. 

The  V&V  process  model,  presented  in  the  following  section,  details  these  activities. 

2.5  Develop  the  Acceptance  Recommendations 

The  final  step  in  the  accreditation  process  develops  the  recommendations  for  the  acceptance  of  the  simulation 
[1],[4],[5],[7]-[9],[12],[17],  These  are  called  the  accreditation  recommendations  when  a  designated  authority 
makes  an  official  decision  to  accept  a  simulation  for  an  intended  use  [4], [8], [9], [12].  Preparing  these 
recommendations  involves  four  tasks: 

•  Analyze  the  validation  evidence. 

•  Develop  the  acceptance  recommendations. 

•  Prepare  the  accreditation  report  and  accreditation  support  package. 

•  Support  the  official  decision  to  accredit. 

Most  authors  who  address  the  accreditation  process  realize  that  the  accreditation  agent  should  analyze  the 
evidence  produced  by  the  V&V  process  to  determine  its  implications  for  the  intended  use  [4],[5]-[9],[17],[18]. 
Acceptance  recommendations  fall  into  five  broad  classes  [2], [4], [5], [12]: 

•  Accept  the  simulation  for  the  intended  use  without  limitations. 

•  Accept  the  simulation  with  limitations  on  the  use. 

•  Defer  the  recommendations  until  further  validation  evidence  is  available. 

•  Defer  the  recommendations  until  the  simulation  is  further  modified. 

•  Reject  the  simulation  for  the  intended  use. 

In  the  many  cases  where  the  simulation  is  recommended  for  the  intended  use  with  limitations,  the  accreditation 
agent  needs  to  identify  the  specific  situations  that  the  users  should  avoid  to  ensure  that  the  simulation  results  will 
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be  sufficiently  complete  and  correct  for  the  intended  use.  Regardless  of  the  recommendations,  the  accreditation 
agent  should  assemble  the  accreditation  plan,  V&V  Plan,  V&V  report,  accreditation  report  and  acceptance 
recommendations  into  an  archival  package  [1], [6], [12], [18]. 

3.0  V&V  PROCESS  MODEL 

From  the  assumptions  above,  the  primary  purpose  for  the  V&V  process  is  to  collect  the  evidence  upon  which 
to  base  the  acceptance  recommendations.  Figure  A2-2  illustrates  the  proposed  model  for  the  V&V  process. 
This  figure  aggregates  the  activities  for  performing  supplementary  verification  and  leveraging  the  verification 
products  into  a  single  box  for  verifying  the  development  products.  Flowever,  these  two  activities  are  discussed 
in  separate  sections  below. 
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Figure  A2-2:  Top  Level  View  of  the  V&V  Process  and  Its  Products. 


3.1  Plan  the  V&V  Effort 

Most  sources  on  VV&A  processes  addressed  V&V  Planning  [2],[4],[5]-[8],[12],[l 7].  The  tasks  for  planning 
the  V&V  effort  mirror  those  for  accreditation  planning  with  one  exception: 

•  Develop  the  V&V  approach. 

•  Build  the  V&V  effort  schedule. 

•  Estimate  the  V&V  costs. 
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•  Prepare  the  V&V  Plan. 

•  Execute  and  evolve  the  V&V  Plan. 

Unlike  the  accreditation  process,  the  V&V  process  does  not  need  a  task  for  collecting  the  V&V  needs  since 
the  accreditation  agent  supplies  those  as  the  accreditation  infonnation  requirements.  As  with  the  accreditation 
process,  the  V&V  practitioner  executes  and  evolves  the  V&V  Plan  throughout  the  V&V  effort.  This  includes 
monitoring  the  V&V  effort  and  adjusting  the  V&V  Plan  to  better  reflect  any  new  information  gained  while 
executing  the  V&V  processes. 

3.2  Apply  Relevant  Historical  Information 

In  those  situations  where  a  legacy  simulation  is  applied  directly  to  a  new  intended  use  or  modified  for  an 
intended  use,  considerable  historical  information  may  exist  upon  which  to  base  acceptance  recommendations. 
In  fact,  the  entire  V&V  effort  may  focus  upon  the  collection  and  analysis  of  historical  information.  Applying 
the  relevant  historical  information  includes  six  tasks: 

•  Collect  and  analyze  the  V&V  and  testing  history. 

•  Collect  and  analyze  prior  use  history. 

•  Collect  and  analyze  developer  accounts  of  simulation  capabilities  and  limitations. 

•  Identify  factors  that  may  constrain  use  from  the  historical  information. 

•  Integrate  the  historical  evidence. 

•  Identify  effective  output  sampling  areas  from  the  integrated  historical  evidence. 

Applying  any  historical  information  begins  by  determining  how  closely  the  prior  intended  uses  match  the 
current  one.  In  effect,  this  decides  what  part  of  the  history  is  relevant  to  the  current  problem.  Historical 
information  can  come  from  prior  V&V  and  testing  activities,  the  records  of  prior  uses,  and  the  developer’s 
accounts  of  the  simulation’s  capabilities  and  limitations.  The  V&V  practitioner  should  then  analyze  the 
relevant  historical  record  to  identify  the  factors  that  may  constrain  use.  Then,  a  coherent  picture  of  the 
simulation’s  capabilities  and  limitations  that  the  historical  evidence  depicts  can  be  assembled.  If  results 
validation  will  be  performed,  the  V&V  practitioner  should  use  the  historical  evidence  to  identify  possible 
output  sampling  areas  to  improve  its  efficiency. 

Only  the  availability  of  relevant  historical  information  can  reliably  substitute  for  results  validation  since  it  will 
include  the  results  of  past  V&V  efforts.  This  is  only  possible  when  the  prior  uses  either  individually  or  in 
combination  cover  the  current  intended  use. 

3.3  Verify  and  Validate  the  Simulation  Conceptual  Model 

Most  of  the  surveyed  sources  recommended  verifying  and  validating  the  simulation  conceptual  model 
[l],[2],[4],[6]-[10],[12],[14],[16],[18]-[20].  Pace  has  described  the  structure  and  content  of  a  prototypical 
simulation  conceptual  model  [21]-[23].  Eight  tasks  are  associated  with  verifying  and  validating  the  conceptual 
model: 
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Characterize  conceptual  model  coverage. 

Check  the  internal  consistency  of  the  conceptual  model. 

Infer  the  intended  representational  capabilities  from  the  conceptual  model. 
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•  Evaluate  the  conceptual  model  validity. 

•  Verify  and  validate  the  available  scenarios. 

•  Identify  factors  that  may  constrain  use  from  the  conceptual  model. 

•  Identify  effective  output  sampling  areas  from  the  conceptual  model. 

•  Integrate  the  conceptual  model  validation  evidence. 

As  with  all  models,  the  simulation  conceptual  model  abstracts  the  functionality  of  the  simulation.  As  a  result,  the 
V&V  practitioner  should  begin  by  characterizing  its  coverage  of  the  intended  use  and  infer  the  simulation’s 
capabilities  from  the  information  that  the  conceptual  model  contains.  In  between  these  tasks,  the  conceptual 
model  verification  involves  checking  it  for  internal  consistency  problems.  Depending  upon  the  detail  of  the 
conceptual  model,  these  tasks  can  vary  from  trivial  to  complex  and  time  consuming.  After  that,  the  V&V 
practitioner  can  evaluate  the  conceptual  model  against  the  acceptability  criteria  to  determine  its  validity. 
If  available,  the  V&V  practitioner  should  also  verify  and,  if  needed,  validate  the  use  scenarios  [8].  The  remainder 
of  the  tasks  parallels  those  in  applying  the  relevant  historical  information. 

As  discussed  earlier,  the  level  of  detail  of  conceptual  models  can  vary  wildly.  If  the  conceptual  model  includes 
the  mathematical  or  theoretical  model  in  addition  to  a  high  level  description  of  the  functionality  of  the 
simulation,  it  can  be  used  to  assess  the  simulation’s  correctness  against  the  validation  referent. 

3.4  Perform  Supplemental  Verification 

This  is  the  first  of  two  development  product  verification  activities.  How  much  the  V&V  practitioner  contributes 
to  development  product  verification  depends  strongly  upon  the  amount  and  quality  of  verification  that  the 
developer  performs  (or  has  performed  for  legacy  simulations).  The  nature  and  degree  of  the  verification  that  can 
be  performed  depends  largely  upon  the  development  products  available  [24],  While  most  sources  recommended 
verifying  the  simulation  design  products  [1], [2], [4], [7], [9], [10], [12], [18]  and  the  implementation  products  or 
executable  model  [1],[2],[4],[8]-[10],[12],[14],[18],[20],  the  developer  may  perform  some  fraction  of  the 
verification  needed  to  support  a  simulation’s  validation.  The  V&V  practitioner  should  leverage  as  much  of  the 
developer’s  verification  as  possible  and  only  perform  what  supplemental  verification  is  needed  to  increase  the 
confidence  in  the  validation  evidence  to  the  desired  degree.  Supplemental  verification  includes  eight  tasks: 

•  Collect  the  developer’s  verification  products. 

•  Determine  the  scope  of  supplemental  verification  needed. 

•  Check  for  computational  anomalies. 

•  Analyze  the  development  products. 

•  Test  the  development  products. 

•  Verify  the  development  products  for  standards  compliance. 

•  Verify  the  interoperability  and  compatibility  of  the  development  products. 

•  Verify  the  development  products  against  the  conceptual  model. 

In  the  tasks  given  above,  the  development  products  cover  both  the  simulation  design  products 
(e.g.,  architectural  design  [1],[6],[7],[24],  formal  model  [20],  detailed  design  specification  [1], [14], [20], [24]) 
and  its  implementation  products  (e.g.,  simulation  code  [2],  interfaces  [8]).  Design  and  implementation  verification 
can  serve  many  purposes  but  the  V&V  practitioner  performs  supplemental  verification  primarily  to  bolster  the 
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validation  evidence.  As  mentioned,  the  V&V  practitioner  begins  the  supplemental  verification  by  collecting 
the  developer’s  verification  products  and  determining  how  much  additional  verification  is  needed  to  achieve 
the  desired  confidence. 

The  next  three  tasks  address  the  types  of  verification  that  the  V&V  practitioner  could  perform  including 
checking  for  computational  anomalies  (e.g.,  stability,  convergence,  representational  errors),  analysis  (e.g.,  control 
sequence,  data  flow)  and  testing  (e.g.,  static,  dynamic).  Often  standards  play  an  important  role  in  simulation 
development  and  the  adherence  to  some  standards  could  significantly  affect  validity.  Thus,  the  V&V  practitioner 
may  need  to  assure  the  compliance  to  those  standards  [6]-[8],[24]  if  the  developer  has  not  already  done  so  to  the 
desired  degree.  Similarly,  the  V&V  practitioner  may  need  to  verify  the  interoperability  and  compatibility  of  the 
implementation  components  both  internally  and  externally  (e.g.,  with  other  simulations  in  a  federation) 
[7], [8], [24],  These  supplemental  activities  produce  evidence  to  assure  that  the  simulation  executable  model  is 
functioning  correctly  and  could,  therefore,  perform  validly.  After  having  gained  confidence  that  the  simulation’s 
execution  is  free  from  error,  the  V&V  practitioner  needs  to  verify  that  the  development  products  faithfully 
reproduce  the  functionality  described  in  the  simulation  conceptual  model.  This  task  links  verification  firmly  to 
the  simulation’s  validity  (limited  by  the  conceptual  model’s  level  of  abstraction)  but  depends  upon  the 
traceability  information  that  exists.  The  detail  of  this  traceability  information  between  the  development  products 
and  the  requirements  through  the  conceptual  model  will  ultimately  determine  what  the  V&V  practitioner  can 
infer  about  simulation  validity  from  the  verification  products. 

The  list  of  tasks  given  above  does  not  include  all  of  the  verification  tasks  that  some  authors  have  suggested. 
One  source  recommended  validating  the  detailed  design  [6]  and  another  recommended  validating  the  executable 
model  [23].  These  steps  were  excluded  because  of  their  difficulty  and  limited  value  to  the  accreditation  process. 
Another  source  included  the  software  quality  and  documentation  assessments  with  the  verification  efforts  [18]. 
Although  this  is  a  reasonable  idea,  these  tasks  are  not  simulation-specific  and  should  be  applied  to  all  software 
regardless  of  its  type. 

3.5  Leverage  the  Verification  Products 

Prior  to  this  activity,  the  V&V  practitioner  has  collected  the  developer’s  verification  products,  assessed  their 
coverage  to  determine  what  supplemental  verification  is  needed  and  performed  that  supplemental  verification. 
In  this  activity,  the  V&V  practitioner  applies  the  collected  verification  information  to  develop  evidence  on  the 
simulation’s  validity,  add  to  the  information  to  support  the  acceptance  recommendations  and  provide  guidance 
for  output  sampling.  This  includes  the  following  six  tasks: 

•  Characterize  the  collective  verification  coverage. 

•  Infer  the  representational  capabilities  from  the  verification  products. 

•  Evaluate  development  product  validity  from  the  verification  products. 

•  Identify  effective  output  sampling  areas  from  the  verification  products. 

•  Identify  factors  that  may  constrain  use  from  the  verification  products. 

•  Integrate  the  verification  evidence. 

In  the  first  task,  the  V&V  practitioner  maps  the  verification  information  onto  the  simulation’s  representational 
space  to  determine  what  of  the  required  functional  inventory  that  information  addresses,  what  parts  of  that 
inventory  are  not  covered  and  what  parts  that  evidence  from  other  sources  (e.g.,  conceptual  model  validation) 
also  considers.  With  the  verification  coverage,  the  V&V  practitioner  can  then  ascertain  the  simulation’s 
representational  capabilities  and  deduce  the  simulation’s  validity  from  the  development  product  verification 
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by  comparing  those  capabilities  against  the  acceptability  criteria.  The  outcomes  from  these  tasks  create  a 
foundation  from  which  to  develop  effective  output  sampling  guidance  and  identify  factors  that  may  constrain 
use.  Finally,  the  V&V  practitioner  needs  to  compile  the  evidence  produced  both  by  analysis  of  the  developer’s 
verification  activities  and  the  supplemental  verification  into  a  coherent  picture  of  the  simulation’s  validity 
painted  from  the  verification  products. 

As  mentioned  earlier,  development  product  verification  can  improve  the  confidence  in  the  entire  body  of 
validation  evidence.  At  this  level,  the  V&V  practitioner  actually  quantifies  the  impact  on  confidence  that  the 
verification  evidence  carries  in  its  contribution.  Estimates  of  the  verification  coverage  of  the  simulation’s 
completeness  and  correctness  can  also  be  used  to  estimate  the  probability  that  any  other  independent  verification 
activities  will  produce  the  same  conclusions.  Correlated  verification  results  that  produce  consistent  conclusions 
about  the  simulation’s  capabilities  improve  the  confidence  in  the  evidence  describing  those  capabilities. 
Likewise,  uncorrelated  results  weaken  the  confidence  in  that  evidence  and  may  require  further  exploration  to 
resolve  any  disagreements.  In  some  cases,  further  verification  can  provide  the  information  to  resolve  these 
problems  and  once  again  improve  confidence  in  the  combined  evidence. 

3.6  Verify  and  Validate  the  Data  and  Knowledge  Sets 

VV&A  requires  analysis  of  both  models  and  data  [14].  The  activities  heretofore  have  concentrated  upon  the 
simulation  model.  This  activity  focuses  upon  the  data  that  is  input  to  that  model  or  upon  which  that  model  sits 
to  derive  its  output  (e.g.,  capabilities  description  files).  Many  of  the  sources  surveyed  emphasized  the 
importance  of  verifying  and  validating  the  data  that  the  simulation  uses  [1], [2], [12], [14].  The  authors  of  this 
paper  distinguish  data  and  knowledge  even  though  some  could  argue  that  knowledge  is  simply  a  type  of  data. 
Knowledge  differs  from  data  when  it  employs  a  specific  knowledge  representation  (e.g.,  production  rules, 
semantic  networks,  neural  networks).  Intelligent  systems  and  human  behavior  representations  commonly 
employ  knowledge  bases  just  as  simulations  of  simpler  physical  phenomena  employ  databases.  The  V&V  of 
knowledge  differs  from  that  for  data  because  it  can  leverage  the  vast  resources  of  techniques,  tools  and 
guidance  from  knowledge-based  system  verification,  validation,  evaluation  and  testing  [25].  Verifying  and 
validating  the  data  and  knowledge  sets  includes  the  following  eight  tasks: 

•  Identify  the  data  and  knowledge  sources  and  their  pedigrees. 

•  Find  authoritative  sources  for  data  and  knowledge  with  none. 

•  Verify  the  internal  consistency  of  data  and  knowledge. 

•  Verify  all  data  transformations. 

•  Validate  data  and  knowledge  sets  where  needed. 

•  Identify  effective  output  sampling  areas  from  the  data  and  knowledge  sets. 

•  Identify  factors  that  may  constrain  use  from  data  and  knowledge  V&V. 

•  Integrate  the  data  and  knowledge  V&V  evidence. 

Data  and  knowledge  set  V&V  should  be  performed  because  of  the  independence  of  those  sets  from  the 
simulation  and  because  the  simulation  relies  upon  those  sets  to  produce  its  output.  Invalid  data  or  knowledge  will 
lead  to  invalid  simulation  results.  A  data  set  essentially  represents  another  form  of  model,  a  model  that  will 
affect  the  validity  of  the  simulation’s  results,  and  therefore  needs  to  be  validated. 

The  tasks  for  data  and  knowledge  V&V  begin  by  identifying  their  sources  and  the  pedigrees  of  those  sources. 
This  takes  into  account  the  fact  that  in  many  cases  the  organizations  responsible  for  producing  the  data  for  a 
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simulation  differ  from  the  organizations  that  either  develop  or  use  the  simulations.  The  credibility  of  those 
data-producing  organizations  often  weighs  heavily  in  determining  the  credibility  of  the  data  itself.  Sometimes, 
data  and  knowledge  comes  to  a  simulation  program  without  any  obvious  pedigrees.  In  those  cases,  the  V&V 
practitioner  needs  to  trace  the  history  of  the  data  (through  configuration  management  documentation  if 
available)  to  identify  the  sources  and  describe  the  authority  of  those  sources.  In  some  instances,  the  V&V 
practitioner  may  need  to  recommend  using  other  data  with  pedigrees  that  are  credible  to  the  users. 

After  establishing  the  pedigrees  of  the  data  and  knowledge,  the  V&V  practitioner  should  verify  the  internal 
consistency  of  the  data  sets,  verify  any  transformations  used  to  make  the  data  accessible  and  meaningful  to  the 
simulation  (e.g.,  units  transformations,  coordinate  transformations)  then  validate  the  completeness  and  correctness 
of  the  data  sets  against  the  acceptability  criteria.  The  V&V  practitioner  can  then  use  the  data  validation 
information  to  identify  output  sampling  guidance  and  factors  that  may  constrain  simulation  use.  The  V&V 
practitioner  then  assembles  the  products  from  the  data  and  knowledge  V&V  into  a  consistent  package  of 
evidence. 

3.7  Validate  Simulation  Results 

Simulation  results  validation  is  almost  universally  included  in  VV&A  process  models  [1],[4], 
[6]-[10],[12],[14],[16]-[18],[20].  In  this  activity  the  simulation  produces  output  through  the  execution  of  test 
scenarios,  the  V&V  practitioner  interprets  the  simulation’s  representational  capabilities  from  that  output  and 
then  compares  those  capabilities  against  the  acceptability  criteria  to  determine  validity.  Results  validation 
involves  seven  tasks: 

•  Plan  for  results  validation. 

•  Leverage  developer  test  results  for  validation. 

•  Collect  simulation  output  for  validation. 

•  Verify  the  simulation  output. 

•  Infer  the  simulation’s  validity  from  its  output. 

•  Identify  factors  that  may  constrain  use  from  results  validation. 

•  Integrate  the  results  validation  evidence. 

On  the  surface,  results  validation  appears  straightforward.  That  would  be  true  if  complete  testing  of  the  simulation 
was  possible  but,  most  times,  complete  testing  is  impractical  and,  many  times,  infeasible  [3].  Therefore,  results 
validation  can  only  sample  from  the  simulation’s  behavior  space  and  the  V&V  practitioner  needs  to  infer  validity 
from  that  limited  sample.  The  first  task,  planning  for  results  validation,  addresses  the  deliberate  choice  of  where 
and  when  to  sample.  This  can  be  done  through  design  of  experiments  techniques  [26]  among  others.  The  previous 
V&V  activities  should  produce  guidance  for  sampling  simulation  output.  The  conditions  of  expected  use  can 
further  constrain  simulation  testing  by  defining  the  boundaries  of  use.  This  guidance  can  make  a  computationally 
intractable  output-sampling  problem  tractable.  The  V&V  practitioner  should  also  use  the  developer’s  test  results 
to  improve  the  efficiency  of  output  sampling. 

Some  authors  suggest  that  the  V&V  practitioner  should  verify  the  output  that  will  be  used  for  results 
validation  [1],[7],[20].  This  task  can  identify  problems  with  the  output  before  expending  the  resources  needed 
for  results  validation.  Finally,  the  V&V  practitioner  needs  to  infer  the  simulation’s  validity  from  its  output, 
identify  factors  that  may  constrain  use  and  integrate  the  results  validation  products  into  a  coherent  evidence 
package. 
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3.8  Integrate  the  Validation  Evidence 

The  preceding  V&V  activities  produce  the  evidence  that  this  activity  integrates  into  a  lucid  description  of  the 
simulation’s  validity  to  support  the  acceptance  recommendations.  This  activity  involves  the  four  tasks: 

•  Infer  simulation  validity  from  collective  V&V  results. 

•  Form  the  validation  conclusions. 

•  Prepare  the  V&V  report. 

•  Support  any  archival  of  the  V&V  products. 

The  V&V  practitioner  should  examine  the  evidence  produced  by  the  preceding  activities,  form  a  consistent 
picture  of  the  simulation’s  validity  and  estimate  the  confidence  in  that  determination  if  needed  and  possible. 
The  V&V  practitioner  can  also  assemble  the  factors  that  may  constrain  use  into  a  single  set  of  use  constraints 
then  combine  those  constraints  with  the  assessments  of  the  simulation’s  completeness  and  correctness  for  the 
intended  use.  Then,  the  V&V  practitioner  assembles  the  results  of  this  integration  with  the  evidence  produced 
by  the  prior  V&V  activities  into  the  V&V  report  [1], [5], [6], [12], [17], [18].  The  information  in  the  V&V  report 
should  present  the  evidence  upon  which  the  accreditation  agent  can  base  the  acceptance  recommendations. 
This  activity  may  include  any  post-execution  follow-up  and  archival  performed  [1],[9]. 


4.0  SUMMARY  AND  CONCLUSIONS 

This  paper  presents  a  comprehensive  model  of  the  VV&A  processes  and  describes  the  activities  and  tasks  that 
could  be  performed  to  meet  a  simulation  program’s  accreditation  needs.  Flowever,  the  authors  do  not  intend 
VV&A  practitioners  to  apply  this  model  in  totality  or  universally.  This  model  only  supplies  the  cloth  from 
which  to  cut  the  VV&A  processes  needed  to  suit  the  demands  and  resource  availability  of  specific  programs. 
The  authors  also  intend  VV&A  practitioners  to  use  their  knowledge  of  the  risks  that  their  simulation  users  can 
and  cannot  accept  in  their  judgments  of  which  VV&A  activities  and  tasks  to  perform  and  at  which  maturity 
level  to  perform  them. 

Research  is  ongoing  to  develop  a  comprehensive  model  of  VV&A  process  maturity  to  accompany  and 
complement  this  VV&A  process  model.  As  Balci  has  observed  “No  rigid  ‘cookbook’  simulation  VV&A 
process  can  fit  all  situations  all  the  time.”  [3]  The  process  model  presented  in  this  paper  is  aimed  at  helping 
VV&A  practitioners  tailor  their  processes  to  best  fit  their  situations. 
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Risk-Based  Accreditation  Methodology:  an  Ontology  Implementation 

(NATO  MSG-054:  “An  Overlay  Standard  for  W&A  of  Federations”) 

Hwa  Feron  Axel  Lehmann 
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An  international  effort  is  currently  under  way  towards  developing  a  Risk-Based  Accreditation  (RBA) 
methodology  for  common  VV&A  frameworks  for  simulations  and  federations  of  simulations.  Its  goal  is  to 
better  manage  resources,  to  be  able  to  account  for  risk  and  to  support  tailoring  in  the  W&A  process.  This  work 
proposes  an  ontological  implementation  of  the  RBA  methodology  structure  cleanly  separating  structure  from 
data  in  order  to  promote  coherent  tailoring,  to  generate  reliable  documentation  and  to  facilitate  future 
methodology  upgrades.  A  few  detailed  examples  were  implemented  with  the  Web  Ontology  Language  (OWL) 
[16]  in  order  to  demonstrate  the  basic  implementation  principles,  however  the  RBA  methodology  is  not  yet 
finalized,  and  neither  are  the  details  of  the  individual  RBA  activities  and  tasks  used  for  those  examples. 


1.0  INTRODUCTION 

The  Risk-Based  Accreditation  (RBA)  methodology  for  common  VV&A  frameworks  for  simulations  and 
federations  of  simulations  is  the  result  of  an  extensive  international  cooperation  and  convergence  of  efforts,  as 
highlighted  by  DMSO  [3],  IEEE  1516.4  [4],  NASA-STD-7009  [9],  NATO  MSG-054  [10],  ND1A  [2],  REVVA 
[12],  S1SO  [13],  etc. 

As  its  name  implies,  RBA  is  organized  to  identify  and  quantify  various  simulation  risks  (as  well  as  resources)  in 
order  to  optimize  the  W&A  process.  Essential  verifications  and  validations  may  thus  be  identified  according  to 
the  importance  of  the  corresponding  risks  and/or  costs.  Those  parts  of  the  VV&A  process  involving  less  risk  are 
liable  to  be  tailored  out  in  order  to  keep  a  lid  on  costs  while  retaining  an  acceptable  level  of  VV&A  reliability. 

The  RBA  methodology  applies  to  the  entire  VV&A  process,  beginning  with  an  Accreditation  Plan  in  which 
the  accreditation  needs  and  resources  are  assessed  in  order  to  develop  a  suitable  accreditation  approach  so  as 
to  determine  optimal  simulation  acceptability  criteria  in  light  of  the  intended  use  and  use  risks.  The  V&V 
needs,  risk,  resources,  schedule  and  cost  resulting  from  this  approach  and  criteria  are  then  determined,  and  if 
appropriate,  the  thus  determined  VV&A  procedure  is  carried  out,  ending  with  an  Accreditation  Decision  with 
relevant  supporting  documents  and  evidence. 

In  other  words,  the  goal  of  the  RBA  methodology  is  to  devise  and  implement  a  VV&A  approach  which 
properly  and  efficiently  determines  the  suitability  (with  respect  to,  e.g.,  cost,  risk,  utility,  scheduling)  of  a 
model,  simulation,  or  federation  of  simulations  for  the  intended  use,  taking  risk  into  account  to  optimize  the 
efficiency  and  reliability  of  this  determination  and  to  adjust  the  complexity  of  the  VV&A  process  in  light  of 
the  available  resources  and  constraints.  In  this  context,  risk  refers  in  particular  to  the  risk  of  using  a  simulation 
(i.e.,  use  risk),  but  risks  inherent  to  the  development  and  VV&A  process  are  also  taken  into  account. 
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After  a  brief  discussion  in  Part  2  of  risk  and  its  assessment  in  VV&A,  Part  3  summarizes  the  current  status  of 
the  international  cooperation  on  the  RBA  methodology,  Part  4  presents  our  implementation  of  RBA  as  an 
ontology  and  Part  5  concludes  the  presentation. 


2.0  RISK  IN  W&A  OF  SIMULATIONS 

2.1  Different  Types  of  Risk 

“Risk  is  a  measure  of  future  uncertainties  in  achieving  program  performance  goals  and  objectives  within 
defined  cost,  schedule  and  performance  constraints.”  [1] 

Risks  associated  with  simulation  development  and  use  can  be  categorized  as  development  risks,  operational 
risks,  inherited  risks,  etc.  [1],[5],[1 1]: 

1)  Development  Risks  (supported  by  program  and  developer  managers)  are  related  to  the  simulation 
development  itself  or  to  the  modification  of  legacy  simulations  and  typically  relate  to  compromises 
made  because  the  simulation  does  not  exactly  meet  the  needs  of  the  intended  application  (e.g.,  inadequate 
representations,  insufficient  accuracy),  or  to  potential  problems  in  addressing  the  technical,  scheduling, 
cost,  or  resourcing  aspects,  or  to  accreditation  activities  modifying  these  aspects  during  the  course  of 
development. 

2)  Operational  Risks  (supported  by  users  and  V&V  practitioners,  managed  by  accreditation  agents)  are 
risks  arising  from  using  simulation  results  that  are  incorrect  and  not  believing  simulation  results  that 
are  correct,  as  well  as  risks  arising  from  V&V  activities  when  minimizing  the  effects  (scheduling, 
budget,  etc.)  of  potential  failures  and  when  producing  the  evidence  of  capability  and  credibility  of 
M&S.  Since  the  V&V  activities  are  meant  to  provide  evidence  for  the  accreditation  phase,  ultimately 
the  accreditation  agent  has  to  manage  operational  risk. 

3)  Inherited  Risks  (especially  when  reusing  models,  for  instance  in  federations)  are  risks  arising  from 
effects  carried  forward  from  previous  simulation  development  or  usage,  such  as  effects  resulting  from 
undocumented  assumptions,  limitations,  and  constraints,  or  errors  and  defects  that  were  either  undetected 
or  considered  insignificant  in  previous  applications. 

This  list  is  not  exhaustive.  Other  important  risk  categories  relate  for  instance  to  the  maintainability  and  reliability 
of  the  simulation. 

2.2  Example  of  Risk  Assessment 

In  order  to  objectively  manage  risk  and  establish  risk-based  priorities  for  VV&A  according  to  the  RBA 
methodology,  it  is  essential  not  only  to  be  able  to  identify  all  or  most  of  the  risks  in  the  preliminary  phase, 
but  also  to  be  able  to  (preferably  quantitatively)  compare  the  different  risks  with  one  another  by  using  an 
established  measure  of  risk  (e.g.,  mathematical  expectation  of  loss)  due  to  an  event,  for  instance  as  follows  [5]: 

(Measure  of  Risk  due  to  an  event)  =  (impact  level)  x  (probability  of  occurrence) 

In  ordinary  management  decisions,  “impact  level”  often  is  the  estimated  monetary  cost  of  the  average  loss 
resulting  from  this  risk,  expressed  either  as  a  numerical  value  or  as  belonging  to  a  predetermined  range  of 
values  identified  by  a  label  (such  as  “high”,  “medium”,  “low”).  Depending  on  the  domain  of  use  and  goals  of 
the  simulation,  other  methods  of  determining  risk  may  be  more  appropriate,  for  instance: 


A3  -2 


RTO-TR-MSG-054 


APPENDIX  3  -  W&A  COMPOSITE  MODEL  VERIFICATION  RESULTS 


*  The  “impact  level”  may  instead  be  a  quantitative  measure  of  potential  loss  of  life  or  a  measure  of 
mission  failure. 

•  The  “probability  of  occurrence”  of  the  event  may  be  unknown  or  only  subjective,  in  which  case  broad 
ranges  of  probability  (such  as  “high”,  “medium”,  “low”)  are  often  used. 


A  detailed  discussion  of  the  various  methods  for  determining  the  different  types  of  risk  in  M&S  is  beyond  the 
scope  of  this  presentation;  for  more  details,  please  refer  to  the  appended  literature  resources  on  risk  definition, 
risk  types  and  risk  assessment. 


3.0  RBA  METHODOLOGY  STRUCTURE 

3.1  RBA  Methodology  Activity  Sequence 

The  RBA  methodology  consists  of  three  major  successive  phases  which  are  represented  by  the  three  columns 
of  Table  A3-1.  The  first  phase,  during  which  Activities  1,  2  and  3  are  meant  to  be  carried  out  in  sequence, 
is  meant  to  efficiently  and  coherently  plan  and  prepare  the  actual  VV&A  effort  as  such,  which  is  subsequently 
conducted  in  Phase  2  (V&V  process  as  per  Activity  4)  and  Phase  3  (Accreditation/acceptance  decision  process 
as  per  Activity  5). 
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Table  A3-1 :  RBA  Methodology  Activity  Overview  (bullets  refer  to  sub-activities)  [10], 


Prerequisite  to  V&V  Process 

V&V  Process 

Accreditation  Decision 

Activity  1 

Plan  Accreditation  Effort: 

•  Assess  the  Accreditation  Needs 

•  Develop  the  Accreditation 

Approach 

•  Identify  the  Accreditation 

Resources 

•  Build  the  Accreditation  Schedule 

•  Estimate  the  Accreditation  Costs 

•  Prepare  the  Accreditation  Plan 

•  Execute  and  Evolve  the 

Accreditation  Plan 

Activity  4: 

Collect  Validation  Evidence: 

•  Plan  the  V&V  Effort 

•  Apply  Relevant  Historical 
Information 

•  Verify  and  Validate  the  Simulation 
Conceptual  Model 

•  Perform  Supplemental  Verification 

•  Apply  the  Verification  Products  to 
Validation 

•  Verify  and  Validate  the  Data  and 
Knowledge  Sets 

•  Validate  the  Simulation  Results 

•  Integrate  the  Validation  Evidence 

Activity  5: 

Recommendations  for  Use: 

•  Analyze  the  Validation  Evidence 

•  Develop  the  Recommendations  for 
Use 

•  Prepare  the  Accreditation  Report 
and  Accreditation  Support  Package 

•  Support  the  Official  Decision  to 
Accredit 

Activity  2 

Characterize  Simulation’s  Use: 

•  Collect  the  User  Needs 

•  Verify  the  Requirements/ 

Objectives 

•  Document  the  Intended  Use 

•  Assess  the  Use  Risks 

•  Define  the  Conditions  of  Expected 
Use 

•  Develop  the  Acceptability  Criteria 

Activity  3 

Construct  Validation  Referent: 

•  Identify  Practical  Referent 
Information  Sources 

•  Collect  the  Applicable  Referent 
Information 

•  Assemble  the  Validation  Referent 

•  Collect  the  Validation  Evidence 

The  RBA  methodology  is  therefore  just  an  overlay  on  traditional  VV&A  methodology  permitting  a  more 
rational  determination  of  acceptability  criteria  and  more  efficient  planning  of  VV&A  while  taking  risk  and 
real-world  constraints  into  account,  at  the  expense  of  a  few  additional  preparatory  steps,  essentially  consisting 
of  Activities  1,  2,  and  3.  The  additional  resource  expenditure  required  by  these  additional  preparatory  steps  is 
offset  by  better  risk  and  resource  control  in  the  overall  process,  a  more  pragmatic  and  resource-effective 
VV&A  process  and  more  useful  and  relevant  acceptability  criteria  yielding  a  more  trustworthy  accreditation. 
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Particulars  of  the  individual  Activities  1-5  are  likely  to  change  and  have  been  specified  in  Table  A3-1  only 
as  a  rough  guide  providing  a  more  specific  understanding  of  the  overall  philosophy  of  the  RBA  methodology 
and  its  influence  on  VV&A. 

3.1.1  Prerequisite  to  the  V&V  Process:  “Activities  1,  2  and  3” 

Activities  1,  2  and  3  are  instrumental  in  planning  the  entire  VV&A  process  and  its  scope. 

Specifically,  Activity  1  (Accreditation  Plan): 

•  Decides  the  scope  of  the  accreditation  effort; 

•  Influences  the  entire  V&V  effort  concerning  V&V  approach,  resources,  schedule,  costs,  V&V  Plan; 

•  Establishes  priorities  for  the  V&V  effort;  and 

•  Identifies  required  information  to  guide  the  accreditation  assessment. 

Activity  2  focuses  on  the  use  risks  assessment  and  on  the  process  of  defining  suitable  acceptability  criteria  in 
light  of  those  risks,  the  constraints  and  the  intended  uses.  Acceptability  refers  to  the  decision  to  apply  a 
simulation  to  an  intended  use,  i.e.,  this  defines  on  which  basis  the  accreditation  decision  will  have  to  be  taken. 

Activity  3  identifies,  collects  and  assembles  the  validation  referent,  against  which  the  simulation  will  be 
measured  and  evaluated. 

3.1.2  V&V  Process:  “Activity  4” 

Activity  4  corresponds  to  the  V&V  steps  of  traditional  VV&A,  but  according  to  RBA  these  are  defined  by  the 
results  and  outcomes  of  Activities  1,  2  and  3,  and  may  be  tailored  according  to  these  outcomes,  thus  taking 
risks  and  resource  availability  into  account.  The  primary  purpose  of  perfonning  V&V,  as  in  traditional 
VV&A,  continues  to  be  the  collection  of  evidence  indicating  that  the  simulation  is  properly  implemented  and 
valid  for  its  intended  use. 

3.1.3  Accreditation  Decision:  “Activity  5” 

Activity  5  results  in  an  Accreditation  Decision  with  supporting  documents,  based  on  the  evidence  from 
Activity  4  according  to  the  acceptability  (acceptance)  criteria  defined  in  Activity  2  (in  the  light  of  risks  and 
resource  availability).  As  in  traditional  VV&A,  the  accreditation  decision  is  a  formal  decision  authorizing  the 
application  of  the  simulation  for  its  intended  use;  it  may  include  recommendations  for  limitations  of  use. 

3.2  Detailed  Tailorable  RBA  Methodology  Examples 

The  RBA  methodology  as  proposed  in  NATO  MSG-054  is  presented  in  the  form  of  an  Excel  sheet  having  the 
format  shown  in  Tables  A3-2  -  A3-5.  It  is  structured  using  the  following  categories  of  items,  or  fields,  which 
will  be  individually  discussed  below: 

1)  Description  of  activities  and  tasks  of  each  activity  (this  includes  the  use  risks  assessment  activity  and 
its  influences); 

2)  Information  required,  information  sources  and  information  interrelations; 

3)  Information  produced  by  each  activity  and  task;  and 

4)  Tailoring  guidance. 
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3.2.1  Description  of  Activities  and  Tasks 

Each  “Activity”  corresponds  to  the  performance  of  a  specific  function  in  the  RBA  methodology  and  normally 
includes  a  sequence  of  several  “Tasks”.  These  “Tasks”  are  assigned  pieces  of  work  which,  unless  tailored  out, 
all  need  to  be  finished  in  order  to  complete  the  activity  to  which  they  belong.  Each  “Task”  requires  information 
from  various  sources  (for  instance  from  previous  completed  tasks)  and  provides  information  for  documentation 
purposes  and/or  for  use  in  subsequent  tasks. 

Table  A3-2  and  Table  A3-3  reproduce  a  fragment  of  this  Excel  sheet  corresponding  to  Task  4. 1.1.1  of  Activity 

4.1.1  corresponding  to  Activity  4.  Although  the  numbering  scheme  is  likely  to  change  as  this  work  progresses, 
this  example  concretely  shows  that  the  RBA  methodology  must  be  implemented  as  a  (tailorable)  sequence  of 
“Activities”. 


Table  A3-2:  Activity  4.1.1  “Develop  the  V&V  Approach”  Description. 


Activity  No. 

Activity  Title 

Activity  Description 

Activity  4 

Collect  the  Validation  Evidence 

The  primary  purpose  of  performing  V&V  is  to 
collect  evidence  indicating  that  the  simulation 
is  valid  for  intended  use. 

Activity  4.1.1 

(This  activity 
corresponds  to 
Activity  4) 

Develop  the  V&V  Approach 

This  activity  selects  the  specific  activities  and 
tasks  that  will  be  performed  as  part  of  the 

V&V  approach 

Table  A3-3:  Tasks  (4. 1.1.1  -  4.1. 1.5)  Description  of  “Activity  4.1.1”. 


Task  No. 

Task  Description 

Task  4.1. 1.1 

Determine  the  V&V  activities  and  tasks  that  could  be  performed 

Task  4. 1.1.2 

Select  level  of  validation  process  maturity 

Task  4.1. 1.3 

Candidate  V&V  techniques 

Task  4.1. 1.4 

Candidate  tool  needed  to  support  V&V 

Task  4. 1.1.5 

Estimate  complexity  of  each  of  the  V&V  tasks  to  be  performed 

3.2.2  Information  Sources  and  Interrelations 

In  order  to  perform  a  task,  information  may  be  required  from  one  or  more  information  sources.  These  belong 
to  two  categories:  “internal  information  sources”  and  “external  information  sources”  defined  as  follows: 

1)  Internal  information  sources  correspond  to  activities  and  tasks:  Information  produced  by  each  activity 
and  task  is  mostly  provided  to  other  subsequent  activities  or  tasks  as  information  input  required  for 
carrying  out  these  other  tasks.  The  fields  for  internal  information  sources  are  therefore  referring  to  an 
activity  or  task  by  the  name  and/or  number  of  that  activity  or  task  providing  the  information. 

2)  External  information  sources  are  entities  such  as  Sponsor,  Developers,  Users,  VV&A  Team,  Test  and 
Evaluation  Team,  etc.  They  are  expected  to  provide  appropriate  information  such  as  V&V  personnel  list, 
simulation  configuration,  management  history,  documentation  from  prior  VV&A  or  testing  activities,  etc. 
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RBA  guidance  from  open  literature  and  standards  information  from  standards  organizations  are  also 
required. 

Table  A3-4  is  an  example  description  of  Task  4. 1.1.1  (which  corresponds  to  Activity  4.1.1),  listing  the 
information  inputs  and  outputs  of  this  task. 


Table  A3-4:  Detailed  Description  of  “Task  4. 1.1.1”. 


Task  No. 

Task  Description 

Information 

Required 

Information 

Source 

Information 

Produced 

Task  4.1. 1.1 

(This  task 
corresponds  to 
Activity  4.1.1) 

Analyze  the 

Accreditation  Plan, 
the  acceptability  criteria 
and  the  RBA  guidance 
to  determine  the  V&V 
activities  and  tasks  that 
could  be  performed 

Accreditation  Plan 

Activity  1.6 

Candidate  V&V 
activities  and 
tasks 

Acceptability 

Criteria 

Activity  2.6 

Use  Risk 

Assessment  Report 

Activity  2.4 

RBA  Guidance 

Open  Literature 

3.2.3  Information  Produced  by  Each  RBA  Activity  and  Task 

Information  produced  by  each  RBA  activity  and  task  must  be  documented  in  a  report  for  documentation  and 
traceability  purposes. 

3.2.4  Tailoring  Guidance 

The  RBA  process,  like  the  resulting  VV&A  process,  is  tailorable. 

“A  general  approach  to  VV&A  tailoring  is  the  selection  and  modification  of  VV&A  activities  to  meet  the 
needs  of  the  intended  use  within  the  risk,  resource  and  implementation  constraints  of  the  program  and  the 
simulation  users  and  processes  as  follows: 

1)  Begin  with  a  composite  model  that  defines  the  entire  spectrum  of  VV&A  activities  and  tasks  that 
could  be  done; 

2)  Select  and  modify  the  components  of  that  model  to  best  satisfy  the  risk  and  resource  constraints  of  the 
simulation  program;  and 

3)  Assemble  the  selected  components  into  a  consistent  process  that  satisfies  the  use  risk  goals  of  the 
simulation  program  while  still  adhering  to  the  policies,  standards  or  guidelines  that  may  be  relevant  to 
the  intended  use.”  [17] 

Since  the  individual  activities  and  tasks  rely  on  information  produced  by  previously  executed  activities  or 
tasks,  ensuring  a  “consistent  process”  when  an  activity  or  task  is  tailored  out  requires  identifying  how  this 
affects  the  remaining  activities  and  tasks,  either  providing  alternative  information  sources  to  those  tasks 
depending  on  the  tailored  one,  or  tailoring  them  out  as  well.  The  corresponding  instructions,  taking  only  the 
first  level  of  those  cascading  dependencies  into  account,  are  currently  documented  for  some  tailorable 
activities  and  tasks  as  “Activity  Tailoring  Guidance”  and  “Task  Tailoring  Guidance”  (see  Table  A3-5). 
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Table  A3-5:  Task  4.4. 1.3,  Task  4.4. 1.4  and  Task  4.4. 1.5  Descriptions. 


Task  No. 

Information 

Required 

Information 

Source 

Information 

Produced 

Task  Tailoring  Guidance 

Task  4.4. 1.3 

Developer’s 

verification 

products 

Task  4.4. 1.2 

Simulation  capabilities 
and  limitations  implied 
by  the  Developer’s 
verification  products 

This  task  only  needs  to  be  performed 
if  the  Developer  has  supplied  the 
products  that  its  verification  activities 
have  produced 

Simulation 

configuration 

management 

history 

Task  4.4. 1.1 

Acceptability 

Criteria 

Report 

Activity  2.6 

Task  4.4. 1.4 

Task  4.4. 1.2 

Gaps  and 

inconsistencies  in  the 
Developer’s 
verification  products 

If  the  Developer  has  produced  no 
verification  products  then  the  set  of 
gaps  consists  of  all  of  the 
acceptability  criteria 

Task  4.4. 1.3 

Task  4.4. 1.1 

Activity  2.6 

Task  4.4. 1.5 

Task  4.4. 1.3 

Estimated  coverage  of 
the  Developer’s 
verification  products 

If  the  Developer  has  produced  no 
verification  products  then  the 
coverage  of  those  products  is  none 

Task  4.4. 1.4 

Activity  2.6 

4.0  RBA  METHODOLOGY  ONTOLOGY  STRUCTURE 
4.1  Overview 

Implementing  the  RBA  methodology  as  a  knowledge  base,  using  an  appropriate  ontology  (formal  specification 
for  a  community  of  agents  of  the  concepts  and  relationships  forming  the  methodology)  would  make  possible 
its  automated  use  by  agents  in  an  intelligent  network.  In  particular,  an  ontology  facilitates  the  implementation 
of  a  reasoning  mechanism  in  order  to  support  tailoring  processes  of  the  RBA  methodology. 

The  use  of  an  ontology  clearly  separating  structure  from  data  and  documentation  in  order  to  implement  the 
RBA  methodology  guarantees  a  greater  flexibility  and  upgradeability  in  case  of  RBA  methodology  changes  or 
updates  (this  is  particularly  important  so  long  as  RBA  is  not  finalized). 

Furthermore,  ontology  representation  tools  [14]]  and  available  search,  navigation,  visualization  and  reasoning 
support  ontology  tools  [15]  seem  particularly  suited  to  provide  the  documentation  and  exhaustive  justifications 
of  tailoring  in  a  clear  graphical  manner  as  needed  for  VV&A. 
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(1)  Conceptualization: 

The  RBA  ontology  modeling  pattern  is  based  on  two  corresponding,  mirror  image  sub-ontologies  named 
“Structure  Description”  and  “Documentation  (Data  Description)”  on  Figure  A3-1  below.  The  former  consists 
of  activities,  tasks  and  their  interrelations  (information  required  and  sources).  The  latter,  as  its  name  implies, 
consists  of  activity  and  task  reports. 


Information  source  &  Information  sink 
(object  property  /  inverse  property) 


Figure  A3-1 :  Overview  of  the  RBA  Methodology  Ontology  Structure. 


(2)  Formalization: 

The  RBA  methodology  was  formalized  as  an  OWL  ontology  [16],  using  Protege  version  3.4  as  ontology 
editor  [14],  and  using  the  essential  OWL  language  features  as  follows  (see  Figure  A3 -2): 

•  Classes; 

•  Object  properties  /  Inverse  properties  (the  latter  being  especially  useful  in  particular  for  the  property 
information  source/sink,  thus  permitting  easy  forward/backward  navigation  through  the  ontology  for 
rapidly  evaluating  the  consequences  of  RBA/VV&A  tailoring); 

•  Data  type  properties;  and 

•  Instances. 
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For  Project:  •  RBAM-OrtoJogy 

Asserted  Hierarchy  ^  ^ 

owtlhng 

►  S  R6AM_Addtcnal_Re<^ijf  ed.tnf  or  matron 

►  O  Modekng  _S»r»Aat»on  Jnsttutron 

▼  0  RBAM.Activty.Task.Documefitsbon 

▼  0  Activty  .Report 

►  a  RActivltyl  .Accredtabon.Ptan.Eftort 

►  a  RActivity2_SiTiulalion_Usabity  .Report 

►  a  RActivity3_Valid«lion.Reterent .Report 

►  a  RAdivity4_Validation_Evkience_Cofection_Report 

►  a  RActrvily_5_Accredtation_Recornmendalions_Report 
►  a  Task. Report 

▼  a  RBAM.Adivty.Task.Structure 

▼  a  Activity 

►  a  ActiMty.l  .Ptan.Accredlation.Ef fort 

►  a  Activ»y_2_Owracterize_S»iiulations_Ltee 

►  a  Activity _3_Construct_ValKlation_Referert 

►  a  Activty  .4  .Cciect.Validalion.Evidence 

►  a  Actrvrty.S.Oevefcp.the.RecommePdations.for.Use 

▼  a  Task 

►  a  Task_A4  I.WPton 

►  a  Task_A4  2_WHstOry 

►  a  Task.A4  3.WCM 

►  a  Task_A4  4_WScpptement 

►  a  T  ask  _A4  7.WSR 

Figure  A3-2:  OWL  Formalization:  Overview  of  the  RBA  Methodology  Ontology. 


4.2  Concept  “Activity” 

The  class  “Activity”  has  sub-classes  (linked  by  “isA”  relations)  of  subordinated  activities.  Each  class  “Activity” 
has  object  and  data  type  properties,  for  instance  as  shown  in  Figure  A3-3. 


T  •  Activity 

►  0  Activity _1  .Plan.Accreditaition.Ef  f  ort 

►  9  Activity  _2_Characterize_Simulations_Use 

►  #  Activity  _3_Construct_Validation_Referent 
T  #  Activity _4_Collect_VaMalion_Evidence 

▼  •  Activity  _4  1  _Plan_W_Ef  tort 

9  Activity  .4.1 .1  .Develop.VV.Approach 
S  Activity _4  1  2_tdemtify_VV_Resources 
•  Activity  .4.1 .3_Build_VV_Schedule 


^  ^  m.  a _ 

■  A4 .1 .1  _hasPart_tasks  (multiple  Task_4 .1 .1 .1  .candidat 
~M  A4.1 .1  .intormation.smk  (multiple  Task_4.1  3.3_vvPlan 
(■  A4 .1 .1  .report  (mu*pte  RA4 .1.1  _WPtan_approach) 

(■  activity  .description  (multiple  string) 

(■) activity .tailoring.tlag  (multiple  owl:oneOt(true  false)) 

■  activity  .tailoring  guidance  (multiple  string) 

■  activity  .version  (mutate  string) 


Figure  A3-3:  OWL  Formalization:  Class  “Activity”,  its  Sub-Classes  and  Properties  of  Class  “Activity  4.1.1”. 
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4.3  Concept  “Task” 

The  class  “Task”  has  sub-classes  (linked  by  “isA”  relations)  of  subordinated  tasks.  Each  class  “Task”  has  object 
and  data  type  properties  for  instance: 


V  •Task 

T  •  Task_A4.1  .WPtan 

V  #  Task_A4  1 .1  .WPtan.Approach 

•  Task_4  1 .1  1  _candidale_W_acJtvties_afxJ_lasks  | 

•  Task_4 1 .1  2_taie<aed_ieve(_oi_va»ci«iion  jjfocat; 

•  Task_4 1 .1  3_candidate_W_lechrK|i»es 

•  Task  .4  1  1  4_candidate_too<_ne«<led_to_suppor1_V 

•  Task_4  1  1 5_estimateti_cofnpiexly  _pe*_W  .tasks. 

►  •  Task_A4  1 2_wPlan_Res<xrces 

►  #  Task_A4 1 3  .wPtsn.Schedule 

►  •  Task_A4  1  A.wPlan.CostEstamatieri 


*  *  ^  &  • _ 

■  A1  6jnfomwrtion_source  (muR^te  Acttvjy.l  6_Prepare_Accr< 

■  A2.4 jntanMttonjaouriM  (mdtple  Adivty_2  4_Ass«ss_Use J 

■  A2  6jnformabon_sourc*  (mijbpte  Adrvly_2  6_Devekop_Acce 

■  RBA_Gc»dance_nlofmalKxi_s«xce  (mutiple  RBA_Guidance) 

■  T4  1  1  1  Jnlof malion_itnk  (muRiple  Task_4  1  1  3_candidate_V' 

■  T4 1  1  1  .report  (rmApte  RT4  1  1  1  .wPtenApproach.cancHdab 
(■  task.descrplion  (mukpte  string) 

■  task  Jatorng. flag  (rwJtpke  owl  oneOi (true  false}) 

■  task  JaRorng^uidarice  (mutpta  string) 

■  task  versron  (multiple  st mg) 


Figure  A3-4:  OWL  Formalization:  Class  “Task”,  its  Sub-Classes  and  Properties  of  Class  “Task  4. 1.1.1”. 


4.4  Concepts  “Activity”  and  its  “Tasks” 

The  relations  (composition)  between  each  class  “Activity”  and  its  corresponding  tasks  (class  “Task”)  are 
defined  by  “hasPart”  relations. 


Activity_4. 1.  l_Develop_W_Approach 

A4. 1 .  l_hasPart_tasks 

Instance* 

Task_4. 1.1.  l_candidate_W_activities_and_tasks 

Task_4- 1.1. 2_selected_level_of_validation_process_maturity 

Task 4. 1.1. 5 estimated comple5!ity per W tasks to be per  formed 

Task 4. 1 . 1 ,4 candidate tool needed to support W 

Task_4. 1.1. 3_candidate_W_techniques 

Figure  A3-5:  Ontology  Visualization:  Class  “Activity  4.1.1”  and  Object 
Property  “A4.1.1_hasPart_  tasks”  (see  also  Figure  A3-3). 


4.5  Concepts  “Information  Source”  and  “Information  Sink” 

An  “Information  Source”  denotes  an  activity  or  task  which  provides  information  to  the  current  activity  or  task. 
An  “Information  Sink”  denotes  an  activity  or  task  which  may  need  information  from  the  current  activity  or 
task.  The  interrelations  of  information  between  (activity  or  task)  and  (activity  or  task)  are  defined  by 
‘^information  source”  as  object  property  and  “  information  sink”  as  its  inverse  property,  in  order  to  permit 
navigation.  This  is  crucial  for  tailoring  needs. 
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Figure  A3-6:  Ontology  Visualization:  Class  “Task  4. 1.1.1”,  “Activity  1.6”  and 
Properties  “A1.6  Information  Source/Sink”  (see  also  Figure  A3-4). 


4.6  Concept  “Information  Produced” 

The  concept  “Information  produced”  is  defined  as  the  property  “  report”  which  links  to  the  class 
“Activity  Report”  or  “Task  Report”.  This  concept  provides  concrete  documentation  for  traceability,  model 
management,  RBA  and  VV&A  purposes. 


0  Activity _Report 

►  0  RActivityl  _Accreditation_Effort_Plan 

►  0  RActivity2_Simulation_Usability_Report 

►  0  RActivity3_Validation_Referent_Report 

►  0  RActivity4_Validation_Evidence_Collection 
0  RActivity5_Accreditation_Recommendations 


Figure  A3-7:  OWL  Formalization:  Class  “Activity  Report”  and  its  Sub-Classes  (see  also  Figure  A3-6). 


4.7  Concepts  “Activity  Tailoring  Guidance”  and  “Task  Tailoring  Guidance” 

The  actual  implementation  of  tailoring  concepts  remains  outside  of  the  scope  of  this  work;  however  the 
example  of  Table  A3-5  and  Figure  A3-8  describes  how  the  concept  “Tailoring  Guidance”  may  be  formalized 
for  Task  4. 4. 1.4  using  the  “informationsource”  object  property.  Specifically,  if  Task  4.4. 1.4  is  tailored  out 
then  the  information  source  to  Task  4. 4. 1.5  is  retrieved  from  Task  4.4. 1.3  and  Activity  2.6  only;  the  “Task 
Tailoring  Guidance”  of  Task  4.4. 1.4,  retrieved  by  tracing  the  properties  “information  source”,  specifies  which 
information  is  to  be  used,  namely  the  default  value  defined  by  “Set  of  gaps  consists  of  all  of  the  acceptability 
criteria”  (instead  of  the  information  “Gaps  and  inconsistencies  in  the  Developer’s  verification  products”  which 
would  have  been  produced  by  Task_4.4.1.4).  Thus  in  the  event  of  tailoring  the  appropriate  information  can  be 
retrieved  from  the  previous  information  sources  by  tracing  the  “information  source”  object  property. 
Alternatively,  in  situations  where  the  use  of  default  values  in  the  tailoring  process  is  deemed  likely  to  produce 
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inaccurate  or  misleading  information,  it  may  be  desirable  to  cause  the  execution  of  the  system  to  stop  with  a 
warning  to  the  user  that  tailoring  is  not  appropriate  in  this  case. 


“Task  4.4.1 .4” 
Before  T ailoring 


“Task  4.4. 1.4” 
After  Tailoring 


Figure  A3-8:  “Task  4.4. 1.4”  Before  and  After  Tailoring  (arrows  represent  the  properties 
“information_source”).  Information  sources  to  “Task  4.4. 1.5”  are  “Task  4.4. 1.3” 
and  “Activity  2.6”  only  after  “Task  4. 4.1. 4”  is  tailored  out. 


An  ontological  approach  makes  it  easier  to  obtain  a  complete  picture  of  these  cascading  dependencies  and 
their  consequences  beyond  the  first  level. 

The  bulk  of  the  remaining  tailorable  tasks  is  not  discussed  because  its  tailoring  guidance  is  trivially 
implementable  (this  task  need  not  be  performed  /  only  needs  to  be  performed  if  some  obvious  condition)  and 
does  not  require  any  tracing. 


5.0  CONCLUSION 

The  implementation  of  the  RBA  methodology  as  an  OWL  ontology  is  feasible  without  making  any  compromise, 
and  appears  to  promise  a  greater  flexibility  and  upgradeability  than  more  conventional  approaches  while 
preserving  the  fundamental  advantages  of  the  RBA  methodology. 

It  will  allow,  with  some  additional  work,  a  seamless  integration  in  comprehensive  ontology  based  modeling 
and  simulation  tool  platforms  under  development  at  ITIS  such  as  “Model  Management  System”  [7]  (including 
the  “Leitfaden  fur  Modelldokumentation”  (Guidelines  for  model  documentation)  [6],  and  VV&A  model 
documentation  [8]. 

Ontological  tools  implementing  reasoning  mechanisms  potentially  can  track  tailoring  effects  throughout  the 
RBA/W&A  sequence  of  activities  and  tasks:  additional  work  might  reveal  whether  sufficient  quality  assurance 
guarantees  can  be  provided  in  this  manner  for  deeper,  more  customized  tailoring  approaches.  Further  work 
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might  also  consider  using  such  tools  to  visualize  tailoring  effects  in  a  user  friendly  manner  in  order  to 
facilitate  tailoring  decisions  and  to  explore  the  relationships  between  risk  constellations  and  the  tailoring 
strategies  most  appropriate  for  them. 
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